From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 115DEC3DA41 for ; Tue, 9 Jul 2024 05:59:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65AAC6B0096; Tue, 9 Jul 2024 01:59:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 60AD06B0099; Tue, 9 Jul 2024 01:59:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AB8F6B009B; Tue, 9 Jul 2024 01:59:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 260CE6B0096 for ; Tue, 9 Jul 2024 01:59:02 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B49F5A4B64 for ; Tue, 9 Jul 2024 05:59:01 +0000 (UTC) X-FDA: 82319160882.23.7B991A5 Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) by imf21.hostedemail.com (Postfix) with ESMTP id 0C10F1C0010 for ; Tue, 9 Jul 2024 05:58:59 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=awWuzFIN; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.171 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720504706; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ybuXPGKJnTRCHaqveoOG9YYSY5DjRvZaZMYAdb94zUA=; b=Kzv71ra7Oazt6c78ROF/VKooRgSUdadCaXFhJltmItFJGYAj3urb898h2RNm/gZVG2xzaV dUEIzVIbUQgXVYQXQHnvicf8tmUWD3kXmiWd5r1lw2LuElMerXvSVjADI/YbOZypNt0vXr qd3xRlyZ4VjcOD+X7N8X9g7CnJNdhjQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720504706; a=rsa-sha256; cv=none; b=ur21T0g0o72WFh8Qdu68tkvhhIeDFfzZgoadWm174DVvDAgIVwxsvhXdj7IS8jAvew9dQM zcyCc3xlARjYMK++CXncO9jy5kTE43WOk0okNNdAiz7EwzLReMbjV9Ku7e85Ax2QoeqH/4 qtwqdVcDxsYFEzLNmc2shHGolYJrXu8= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=awWuzFIN; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.171 as permitted sender) smtp.mailfrom=yuzhao@google.com Received: by mail-qt1-f171.google.com with SMTP id d75a77b69052e-447df43324fso141981cf.1 for ; Mon, 08 Jul 2024 22:58:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720504739; x=1721109539; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ybuXPGKJnTRCHaqveoOG9YYSY5DjRvZaZMYAdb94zUA=; b=awWuzFINibP2v8ERZ0K9lok8KB9HOb/ZeK8P5Y9lu3voJlbPRzxD808GP+dTQ3FxXp IxOSbMeyMkk4jo6xTIEJjo6RQyRC3y7iVdvrdRKVw155PWP9agvFQOyMbNNCENhFm5eU W9tz92Rt5Xf0a95jYFR/THDIY6W5ad0bw3TxOdemPdHNdD6rMboa3GPucBhLprqa0EyY plV2qivO++LBKQfFT4xFVGn/5j7qeSxaZfopkayWG177NxYLIa56CqtAj8dJhFNmXd4V zbJ6zb/6D2zFrPX2E2CMV3HzvUnHpue9L1EDUNuWjbI8SJYWq6YG5wwbH4jiJTtZQ1O/ ghfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720504739; x=1721109539; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ybuXPGKJnTRCHaqveoOG9YYSY5DjRvZaZMYAdb94zUA=; b=XtcBuuzqPd/SjMVESTQJS44kKe0N+0BRZnR064puDzWt9jP+aeyoV6RMg4rk3ZTh+7 +NT5xIuTt2fJEPc6HvE3yMnF3KRer/0asXsH+KQIBKIjRDa8BhhTVRAQD98fyWRtM5kg eg+k6nRvCiVhLjUR1VtUzQMAPCEBzeLPkVxZlpJ+TYej9ijRJuF6bBsjjEqD0rr0YPKZ oAbcwqMGWDlCn3n0WxF8sf7pbAzDbfk/OTQ6/wHjT0g5mj/fGKiIr2YRa8h3X0RGKT7Q MzSbN3nAKZUtFMsxiCs3umlqgDIXijB4g+bs84BF4WpsIt5LQ8dXDTojIS4GALwNHB9H K9gQ== X-Gm-Message-State: AOJu0Yy8kCJMqgwEOp9hk5vxUR6Ton4oJTMa5FkdVe33R4XUb9voGgc6 6eSdVEgB0R9Mf8AUYWYKp0boWBBgqOPydBxodXslWrZhaDyfXTlCGKKQC3n/77dMrRCOOSkD07u 3uA/vqQv89OitjkhnACF/cmMXyxEAjVZT53AB X-Google-Smtp-Source: AGHT+IE9s6EeXXa5+5jH0FLBqqwcSIlaabORpUN/0MoGtzOnz56BqLqdzYocfeDKJEgc0y/IWf/mYDF4wkludJ0+4q8= X-Received: by 2002:ac8:4dcd:0:b0:447:ed5f:bca6 with SMTP id d75a77b69052e-447fcfe9339mr1355951cf.5.1720504738887; Mon, 08 Jul 2024 22:58:58 -0700 (PDT) MIME-Version: 1.0 References: <1998d479-eb1a-4bc8-a11e-59f8dd71aadb@amd.com> <7a06a14e-44d5-450a-bd56-1c348c2951b6@amd.com> In-Reply-To: <7a06a14e-44d5-450a-bd56-1c348c2951b6@amd.com> From: Yu Zhao Date: Mon, 8 Jul 2024 23:58:21 -0600 Message-ID: Subject: Re: Hard and soft lockups with FIO and LTP runs on a large system To: Bharata B Rao Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, nikunj@amd.com, "Upadhyay, Neeraj" , Andrew Morton , David Hildenbrand , willy@infradead.org, vbabka@suse.cz, kinseyho@google.com, Mel Gorman Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0C10F1C0010 X-Stat-Signature: a3148o7nd33fr65iwsqumnzyhmnbrwtq X-Rspam-User: X-HE-Tag: 1720504739-916067 X-HE-Meta: U2FsdGVkX19efZ/xs2HUlyv9BXfXvXNH246pF7DPwGXIE9TxFxJ5ImLpGvC5Rzas7sZOI6pe7fchTJIY++1OKqQEA9niJEnlLOj3E3OL6NkLpjXAalZ3mG0sDOcDfHt7LNU/GN8qo1/ONzS5xXMMTuKluXUj0kbbJFAZ7fEI+4V1BT48zFpDJwxn1SFD7dmmbA7AVUV6VuifKhJTzmFk2k8nFERDiRCoV8URhK2p4nG2gWLTJmQRd1sCJv6VyTUejKLWLkHIhWdA03VaPIYn6Oo71DBRQoSCXsovRegcEsuW5LAyxRErMNgT3IHTsZK+9W1D9LKKj27V4GUF0o0EYpzUOay4rIm+owmEfQGBOb705B2015jryuDYM5kcBeNAWhMnnK5Hqir5Xh1W8zcT8jWxcJNJoBdwgSTGqIGBP2cRPx93il9Nh8B2boPXx+2qCbQ2AOfnJf3QMCbkrYQllrqY/VrSudKFy0GQ6iU+8UZtfyXUlzkoe/eTUG3uap62FIflrPrPXFEkXFskjtwkrkuGKtLPkYkWUtqFdfhsURrugKkYDM6/zaKgUdgqMxYEm64c6kqtchwCbMgTi68hKEWpasjRVtFdigQJibc9unZ7GNlqyldwF0ZEelxBalHJI7zdgkPYgJmtCpBr68QKLqrKWw8Dn8HIUs3/bxIMq58/c4N/PeUfgmzft3ZlhF215L86ias8DTcKIMXVjoVbtFr8NBvFXmCPMMPP6WlU+ojnG8QhxJeS1Plh9gSzrJuXLLeIyxS+oYGMuOuS8xMpc5bbM0L6OEPv5AIFSlvsCZuJG38+OAtu5NoDFu0mcOVzd76hfGC+Zcm+cRHyjjQIY2LB8sZCjnsHSVxV0YjLoEJPhxjOZg3/8VIH+aq+6mFhj/GeRhkEojXSRJcx43xBbIbJGSOh152+9AEwcFCe5qSqvztzqN0H7fdD6logTmJcpLPRnZxiD/3G2DS3bgg vovNO6IE Qrv9feaC3SP5E7tTY1Iw5OA1hAHJCSyKGjddW9ZhaKG8AJ3Wyur1F9O78IVjAjZSw5Hb5e7a1iDQrUZ5U3khksTXqxOsOdhfeSoJogp1paVKDO1q98ZI65njIjJFM3+hk1p/Fxlz2C3/45mnqMJmqClHshB/vE+qCnpn1xs2KWdLznsOKMI76Suewl6SoW1SfXooT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jul 8, 2024 at 10:31=E2=80=AFPM Bharata B Rao wro= te: > > On 08-Jul-24 9:47 PM, Yu Zhao wrote: > > On Mon, Jul 8, 2024 at 8:34=E2=80=AFAM Bharata B Rao = wrote: > >> > >> Hi Yu Zhao, > >> > >> Thanks for your patches. See below... > >> > >> On 07-Jul-24 4:12 AM, Yu Zhao wrote: > >>> Hi Bharata, > >>> > >>> On Wed, Jul 3, 2024 at 9:11=E2=80=AFAM Bharata B Rao wrote: > >>>> > >> > >>>> > >>>> Some experiments tried > >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >>>> 1) When MGLRU was enabled many soft lockups were observed, no hard > >>>> lockups were seen for 48 hours run. Below is once such soft lockup. > >>> > >>> This is not really an MGLRU issue -- can you please try one of the > >>> attached patches? It (truncate.patch) should help with or without > >>> MGLRU. > >> > >> With truncate.patch and default LRU scheme, a few hard lockups are see= n. > > > > Thanks. > > > > In your original report, you said: > > > > Most of the times the two contended locks are lruvec and > > inode->i_lock spinlocks. > > ... > > Often times, the perf output at the time of the problem shows > > heavy contention on lruvec spin lock. Similar contention is > > also observed with inode i_lock (in clear_shadow_entry path) > > > > Based on this new report, does it mean the i_lock is not as contended, > > for the same path (truncation) you tested? If so, I'll post > > truncate.patch and add reported-by and tested-by you, unless you have > > objections. > > truncate.patch has been tested on two systems with default LRU scheme > and the lockup due to inode->i_lock hasn't been seen yet after 24 hours r= un. Thanks. > > > > The two paths below were contended on the LRU lock, but they already > > batch their operations. So I don't know what else we can do surgically > > to improve them. > > What has been seen with this workload is that the lruvec spinlock is > held for a long time from shrink_[active/inactive]_list path. In this > path, there is a case in isolate_lru_folios() where scanning of LRU > lists can become unbounded. To isolate a page from ZONE_DMA, sometimes > scanning/skipping of more than 150 million folios were seen. There is > already a comment in there which explains why nr_skipped shouldn't be > counted, but is there any possibility of re-looking at this condition? For this specific case, probably this can help: @@ -1659,8 +1659,15 @@ static unsigned long isolate_lru_folios(unsigned long nr_to_scan, if (folio_zonenum(folio) > sc->reclaim_idx || skip_cma(folio, sc)) { nr_skipped[folio_zonenum(folio)] +=3D nr_pages; - move_to =3D &folios_skipped; - goto move; + list_move(&folio->lru, &folios_skipped); + if (spin_is_contended(&lruvec->lru_lock)) { + if (!list_empty(dst)) + break; + spin_unlock_irq(&lruvec->lru_lock); + cond_resched(); + spin_lock_irq(&lruvec->lru_lock); + } + continue; }