From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DF88C3DA41 for ; Wed, 10 Jul 2024 18:05:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC6556B0096; Wed, 10 Jul 2024 14:05:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D76266B009A; Wed, 10 Jul 2024 14:05:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3E506B009C; Wed, 10 Jul 2024 14:05:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A4D066B0096 for ; Wed, 10 Jul 2024 14:05:28 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1E0D21A01F1 for ; Wed, 10 Jul 2024 18:05:28 +0000 (UTC) X-FDA: 82324620336.19.FDA9DCC Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf23.hostedemail.com (Postfix) with ESMTP id 41563140011 for ; Wed, 10 Jul 2024 18:05:26 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ghaRNMYE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720634690; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rOvbHMK6yjy+j4wGCOQ884vrN5y0/inUH90JR72cu7w=; b=3e6Ki/CwSx2/Gkzg6UxqFIJNzub83WxS8dnjl74QKODTYgOfWbWt4hL0RZTl2NJxT0Kv6o nT2qTOxG+hkqJu6u8zAdWTJvej6VEDpI7G8yzKRyaV4BbFREAdQletnge31OAQOK0nvXpy nPmRWTRqEUBEVGQ6unAiQk83TTNJCoI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720634690; a=rsa-sha256; cv=none; b=LhILJjtuzgwSvcgbzqke/ldCOaTUZYtwdvEEPZ3KFIOCq1R9Qcep/YvTZveqkbgQ0WwNu9 3zUs1TZAQ0Y8jJyv3YOGz+NM37JDUM5xPHONQaNgXdaSqjPt4RpJhePZvGi85oD7+5zt87 qajvWi0ansxige4G4VKc/PP0LxSCC+A= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ghaRNMYE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=yuzhao@google.com Received: by mail-qt1-f173.google.com with SMTP id d75a77b69052e-447f8aa87bfso43331cf.0 for ; Wed, 10 Jul 2024 11:05:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720634725; x=1721239525; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=rOvbHMK6yjy+j4wGCOQ884vrN5y0/inUH90JR72cu7w=; b=ghaRNMYExbYoC/SaEziBIaUakXkFYsxW57/2v0wdFafWrTmJH/XzjNv2TiPUt8xVCL 9+OSPedXBmDU6KUVHph75jT0VUfmlM2cG6SrPcunBRvXkKUgj0GLmLcemV0UmoTxzAbU jfTNGCLKfLRvTtku1XTvCKRdN84V1ri7wpWtuhw9IsleYvhClbWjgbciEZVxQh6tYICT l5L5xIxGXBWeQobh8VsVvE9eJ3Qb7+6t6fFvlb+H+GKFdUEGVWlPCXWd7COS7SATNMvD foEiu7lBHUjk+tAGT3uu7Vozrs6kGN4pDvLb2q9PqQIffODVJHmOyEXfCq17aku+laBV VPbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720634725; x=1721239525; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rOvbHMK6yjy+j4wGCOQ884vrN5y0/inUH90JR72cu7w=; b=lb1dKHPOdyD3mehZd8mlwjojTrz27wrDCLNBX+3RhGRA0GgwonVQYo4gQnH6xV9eCX gdrghogV6VHPN8IFEh4/lo/Zs8hLSHjUAHCu2MiXzRZVW4e231JKUhehUFD9uMI9CZMg tUOJ3XsSoEsETHcVZfcQGkkPlYfX8qCvEa/0AOEelipTBYsYHnDEp5Gk5D4U1AfCk9tE m1LfcoXfckgWbsmECCja2+dMLTQqNMcBZ9gUP8XMrUZKUyDNGZxzQpNXe/NDbqWe9J63 HvJCNTwjYbrP7BPBcW2b3IUY1rUACyNVXmOI4fyk+znk5JTCPgi+ylcRV5EP+bYGv5YT Pj/g== X-Forwarded-Encrypted: i=1; AJvYcCWPKQk9/bu8BTU+Yhrkf4naESjEnhrhp5fyaSoTboO1Q+AZwGiAtt6rpI8n0YoEc+TbKiKLhRdcfDH7XcLtyEBHSz8= X-Gm-Message-State: AOJu0YwjvxwZG7GWwDzvraLlx9KfFOBR29EphThYUasszEx9uWTYLrG6 Ebtw4Lo2683LcNNDdXCu5izgbb4+N3mnVupCc1ldbk0CiskKjqKAUg12+B0Z6zdwQ91CmQMfdLg PYMjdNoRjxij+4nNCjePDOcojztyw2n15CoJd X-Google-Smtp-Source: AGHT+IEyytpgDqK3ENrH4AkMFMoYmBex4aMvgkYoPQubRHHsNpCGrYxuusMYiUPUaqGeonOe2ZYTFL/k1eo/+mBo/VM= X-Received: by 2002:a05:622a:7602:b0:447:e497:95d0 with SMTP id d75a77b69052e-44d11e8a35fmr39911cf.17.1720634725183; Wed, 10 Jul 2024 11:05:25 -0700 (PDT) MIME-Version: 1.0 References: <4307e984-a593-4495-b4cc-8ef509ddda03@amd.com> In-Reply-To: <4307e984-a593-4495-b4cc-8ef509ddda03@amd.com> From: Yu Zhao Date: Wed, 10 Jul 2024 12:04:48 -0600 Message-ID: Subject: Re: Hard and soft lockups with FIO and LTP runs on a large system To: Bharata B Rao Cc: mjguzik@gmail.com, david@fromorbit.com, kent.overstreet@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, nikunj@amd.com, "Upadhyay, Neeraj" , Andrew Morton , David Hildenbrand , willy@infradead.org, vbabka@suse.cz, kinseyho@google.com, Mel Gorman , linux-fsdevel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 41563140011 X-Stat-Signature: cxj85rwe41wasjhgiew8fam39woptioc X-Rspam-User: X-HE-Tag: 1720634726-636970 X-HE-Meta: U2FsdGVkX1/ysOGuRKgxlX8XVA4AznnoeEDhu/P/4mOQGwZyWQFw0o+tPmPVzyhgLKkDPQ8OZJysIhawGrVsX8+nqEnhV16soV7fSDdjpA/MUQ5FGjnVSvUx71Ncfr/XmKlGlZRBYwp2p/XEETrKepxxRfMjRn0Go9IuxrjReRorH1fnwsCuRGwHrueUw2Cf1erBcoq0l4BcYVNu9Uc9HYg3ktxE+oCHE47UJ/fZYUOvV+bBjndPjZmYMPGsh/hfGSxlS3/1ebvW1txp2XyvLWeVIzFQM7XIS7rifoi1qml+fmsMIBpfqlufEtOROb5oVyuTAR4BUyXh5KhoJZRnMSEtXDoN8wBzvDeUd6rMMYYTnrLJS+Ha6u7UBF67q2KLufvvOVpx8oO2jQo08uHukXs/EJ1bb6pWBv6M81SDGsT/wc0Q9k7prVjZCBLsDFLA3AjhNiBhXJmZKfVZASv+C7sjGs+onoWVMeIY8CBUNhqMa6eBV6VCruRuxmsmXQl1MGiTNCDxbJ+hKzQd6TXVy4Uox3/tOlziAKoNfW6ailvN+QueR2vUdBnvb6CtdQPhWAGbiGpexIH4iRx53RbPMSR3Dw3EkDxtunTmkc7ZXIzG1YjRI9BZYEgUzHRfI1mHzS0hoQvMvRABJgQOZ4QH/FVotLRpqRkWBM1MspebPQxgUmCRSHlAV/OVfdzmBp8l/EGV8KHB4japfxtHAsmg74DET+4H5WoScxlOJXNS5FQpEbhHyM+do/S1rR3tr4rwLbTvnWVCBc+EgbeKQvhVYWrHIZus7qM0oRMlVvh5bJImGP7BeKIp56BPiCGaKfOxoj/Iwp9ekLuMgYpJF0H70nxdzTv1kzNaE2sCcqBmntTxYmFD4X3BY03hiZALt3ZquiwAyqM/b86U+zkdgjGtHK+VDcEqppttCWxgs6lu7UPx40xMr57vSckROwfazyiQToAivW/6mJUKn9UsBsQ yA+WkWtI q7FSCldgPdV6qQH6cB8dGo7pEA2KNMj+IlzkY+nh1DHAcSUNlRiZnwN/Ls2cCHsYRqQGf9uDqmoR3CZJVoHxTZXlXUk8TQGXKdmGitf+6iKV80YuMCF9buZBGyNRWZqsFablITDAaRSr8Yk1nsU5K3WdkYMmEGevR+E/kGyRMefdPo7YtnVYhSfDN5uJh+iq9Lj0zFFIzSb4O1GImOhy5k3jv3NgFtbmvy0fxhzpgAgGc37buytFqEuVUg1IcgNy7sgkNfYbmDBjOWHNRj4u6ilnRJa/XjvTMBMYv68K+f4R5HY9qguXkk2YNAZa5zMgNryaJvb6zjPYXvgZC+JngeQNGQw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000019, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jul 10, 2024 at 6:04=E2=80=AFAM Bharata B Rao wro= te: > > On 07-Jul-24 4:12 AM, Yu Zhao wrote: > >> Some experiments tried > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> 1) When MGLRU was enabled many soft lockups were observed, no hard > >> lockups were seen for 48 hours run. Below is once such soft lockup. > > >> Below preemptirqsoff trace points to preemption being disabled for mor= e > >> than 10s and the lock in picture is lruvec spinlock. > > > > Also if you could try the other patch (mglru.patch) please. It should > > help reduce unnecessary rotations from deactivate_file_folio(), which > > in turn should reduce the contention on the LRU lock for MGLRU. > > Thanks. With mglru.patch on a MGLRU-enabled system, the below latency > trace record is no longer seen for a 30hr workload run. Glad to hear. Will post a patch and add you as reported/tested-by. > > > >> # tracer: preemptirqsoff > >> # > >> # preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-mglru-irqstr= c > >> # --------------------------------------------------------------= ------ > >> # latency: 10382682 us, #4/4, CPU#128 | (M:desktop VP:0, KP:0, S= P:0 > >> HP:0 #P:512) > >> # ----------------- > >> # | task: fio-2701523 (uid:0 nice:0 policy:0 rt_prio:0) > >> # ----------------- > >> # =3D> started at: deactivate_file_folio > >> # =3D> ended at: deactivate_file_folio > >> # > >> # > >> # _------=3D> CPU# > >> # / _-----=3D> irqs-off/BH-disabled > >> # | / _----=3D> need-resched > >> # || / _---=3D> hardirq/softirq > >> # ||| / _--=3D> preempt-depth > >> # |||| / _-=3D> migrate-disable > >> # ||||| / delay > >> # cmd pid |||||| time | caller > >> # \ / |||||| \ | / > >> fio-2701523 128...1. 0us$: deactivate_file_folio > >> <-deactivate_file_folio > >> fio-2701523 128.N.1. 10382681us : deactivate_file_folio > >> <-deactivate_file_folio > >> fio-2701523 128.N.1. 10382683us : tracer_preempt_on > >> <-deactivate_file_folio > >> fio-2701523 128.N.1. 10382691us : > >> =3D> deactivate_file_folio > >> =3D> mapping_try_invalidate > >> =3D> invalidate_mapping_pages > >> =3D> invalidate_bdev > >> =3D> blkdev_common_ioctl > >> =3D> blkdev_ioctl > >> =3D> __x64_sys_ioctl > >> =3D> x64_sys_call > >> =3D> do_syscall_64 > >> =3D> entry_SYSCALL_64_after_hwframe > > However the contention now has shifted to inode_hash_lock. Around 55 > softlockups in ilookup() were observed: This one is from fs/blk, so I'll leave it to those experts. > # tracer: preemptirqsoff > # > # preemptirqsoff latency trace v1.1.5 on 6.10.0-rc3-trnmglru > # -------------------------------------------------------------------- > # latency: 10620430 us, #4/4, CPU#260 | (M:desktop VP:0, KP:0, SP:0 HP:0 > #P:512) > # ----------------- > # | task: fio-3244715 (uid:0 nice:0 policy:0 rt_prio:0) > # ----------------- > # =3D> started at: ilookup > # =3D> ended at: ilookup > # > # > # _------=3D> CPU# > # / _-----=3D> irqs-off/BH-disabled > # | / _----=3D> need-resched > # || / _---=3D> hardirq/softirq > # ||| / _--=3D> preempt-depth > # |||| / _-=3D> migrate-disable > # ||||| / delay > # cmd pid |||||| time | caller > # \ / |||||| \ | / > fio-3244715 260...1. 0us$: _raw_spin_lock <-ilookup > fio-3244715 260.N.1. 10620429us : _raw_spin_unlock <-ilookup > fio-3244715 260.N.1. 10620430us : tracer_preempt_on <-ilookup > fio-3244715 260.N.1. 10620440us : > =3D> _raw_spin_unlock > =3D> ilookup > =3D> blkdev_get_no_open > =3D> blkdev_open > =3D> do_dentry_open > =3D> vfs_open > =3D> path_openat > =3D> do_filp_open > =3D> do_sys_openat2 > =3D> __x64_sys_openat > =3D> x64_sys_call > =3D> do_syscall_64 > =3D> entry_SYSCALL_64_after_hwframe > > It appears that scalability issues with inode_hash_lock has been brought > up multiple times in the past and there were patches to address the same. > > https://lore.kernel.org/all/20231206060629.2827226-9-david@fromorbit.com/ > https://lore.kernel.org/lkml/20240611173824.535995-2-mjguzik@gmail.com/ > > CC'ing FS folks/list for awareness/comments. > > Regards, > Bharata.