From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F5DFC433EF for ; Thu, 3 Feb 2022 10:12:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 950418D013F; Thu, 3 Feb 2022 05:12:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 900768D0129; Thu, 3 Feb 2022 05:12:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7EF1D8D013F; Thu, 3 Feb 2022 05:12:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0196.hostedemail.com [216.40.44.196]) by kanga.kvack.org (Postfix) with ESMTP id 70B118D0129 for ; Thu, 3 Feb 2022 05:12:32 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 2A8D1824C420 for ; Thu, 3 Feb 2022 10:12:32 +0000 (UTC) X-FDA: 79101054144.19.8A7265A Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf06.hostedemail.com (Postfix) with ESMTP id AE46C180004 for ; Thu, 3 Feb 2022 10:12:31 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id A5B011F3A5; Thu, 3 Feb 2022 10:12:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1643883150; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Gr3C28hfRKIsSogmgFmAlwO97s2JhEgCbYXvCeygykg=; b=s8loehmza/5APLpfD591EuyiwIjmdCM4AHB+n6QqgJ+0PH+jV5VDVQVpxdzvIu5v8rX1/E nDr92yW4IhuSgZVdbUlSZs5H7U352e/+njt4bxJ06M8tSVMVXWDUm0qVN7V5/UUCyUDBwu 4sVTiAt+6VnT9u5oFovnjL1ra46hFOE= Received: from suse.cz (unknown [10.100.201.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 74138A3B87; Thu, 3 Feb 2022 10:12:30 +0000 (UTC) Date: Thu, 3 Feb 2022 11:12:29 +0100 From: Michal Hocko To: Mel Gorman Cc: Andrew Morton , Hugh Dickins , Vlastimil Babka , Rik van Riel , Linux-MM , LKML Subject: Re: [PATCH] mm: vmscan: remove deadlock due to throttling failing to make progress Message-ID: References: <20220203100326.GD3301@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220203100326.GD3301@suse.de> X-Rspamd-Queue-Id: AE46C180004 X-Rspam-User: nil Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=s8loehmz; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf06.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com X-Stat-Signature: sutngix89f8fwc9si3xyn9dh1x6ahorz X-Rspamd-Server: rspam08 X-HE-Tag: 1643883151-403733 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu 03-02-22 10:03:26, Mel Gorman wrote: > A soft lockup bug in kcompactd was reported in a private bugzilla with > the following visible in dmesg; > > [15980.045209][ C33] watchdog: BUG: soft lockup - CPU#33 stuck for 26s! [kcompactd0:479] > [16008.044989][ C33] watchdog: BUG: soft lockup - CPU#33 stuck for 52s! [kcompactd0:479] > [16036.044768][ C33] watchdog: BUG: soft lockup - CPU#33 stuck for 78s! [kcompactd0:479] > [16064.044548][ C33] watchdog: BUG: soft lockup - CPU#33 stuck for 104s! [kcompactd0:479] > > The machine had 256G of RAM with no swap and an earlier failed allocation > indicated that node 0 where kcompactd was run was potentially > unreclaimable; > > Node 0 active_anon:29355112kB inactive_anon:2913528kB active_file:0kB > inactive_file:0kB unevictable:64kB isolated(anon):0kB isolated(file):0kB > mapped:8kB dirty:0kB writeback:0kB shmem:26780kB shmem_thp: > 0kB shmem_pmdmapped: 0kB anon_thp: 23480320kB writeback_tmp:0kB > kernel_stack:2272kB pagetables:24500kB all_unreclaimable? yes > > Vlastimil Babka investigated a crash dump and found that a task migrating pages > was trying to drain PCP lists; > > PID: 52922 TASK: ffff969f820e5000 CPU: 19 COMMAND: "kworker/u128:3" > #0 [ffffaf4e4f4c3848] __schedule at ffffffffb840116d > #1 [ffffaf4e4f4c3908] schedule at ffffffffb8401e81 > #2 [ffffaf4e4f4c3918] schedule_timeout at ffffffffb84066e8 > #3 [ffffaf4e4f4c3990] wait_for_completion at ffffffffb8403072 > #4 [ffffaf4e4f4c39d0] __flush_work at ffffffffb7ac3e4d > #5 [ffffaf4e4f4c3a48] __drain_all_pages at ffffffffb7cb707c > #6 [ffffaf4e4f4c3a80] __alloc_pages_slowpath.constprop.114 at ffffffffb7cbd9dd > #7 [ffffaf4e4f4c3b60] __alloc_pages at ffffffffb7cbe4f5 > #8 [ffffaf4e4f4c3bc0] alloc_migration_target at ffffffffb7cf329c > #9 [ffffaf4e4f4c3bf0] migrate_pages at ffffffffb7cf6d15 > 10 [ffffaf4e4f4c3cb0] migrate_to_node at ffffffffb7cdb5aa > 11 [ffffaf4e4f4c3da8] do_migrate_pages at ffffffffb7cdcf26 > 12 [ffffaf4e4f4c3e88] cpuset_migrate_mm_workfn at ffffffffb7b859d2 > 13 [ffffaf4e4f4c3e98] process_one_work at ffffffffb7ac45f3 > 14 [ffffaf4e4f4c3ed8] worker_thread at ffffffffb7ac47fd > 15 [ffffaf4e4f4c3f10] kthread at ffffffffb7acbdc6 > 16 [ffffaf4e4f4c3f50] ret_from_fork at ffffffffb7a047e2 > > The root of the problem is that kcompact0 is not rescheduling on a CPU > while a task that has isolated a large number of the pages from the > LRU is waiting on kcompact0 to reschedule so the pages can be released. > While shrink_inactive_list() only loops once around too_many_isolated, > reclaim can continue without rescheduling if sc->skipped_deactivate == > 1 which could happen if there was no file LRU and the inactive anon list > was not low. I guess it should be mentioned explicitly that the problem is !PREEMPT specific. Other than that looks good to me. > Debugged-by: Vlastimil Babka > Signed-off-by: Mel Gorman Acked-by: Michal Hocko > --- > mm/vmscan.c | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 090bfb605ecf..59b14e0d696c 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -1066,8 +1066,10 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason) > * forward progress (e.g. journalling workqueues or kthreads). > */ > if (!current_is_kswapd() && > - current->flags & (PF_IO_WORKER|PF_KTHREAD)) > + current->flags & (PF_IO_WORKER|PF_KTHREAD)) { > + cond_resched(); > return; > + } > > /* > * These figures are pulled out of thin air. -- Michal Hocko SUSE Labs