From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 207CFC47E49 for ; Wed, 23 Oct 2019 07:12:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C5C9621D71 for ; Wed, 23 Oct 2019 07:12:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C5C9621D71 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=fromorbit.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 785C06B0007; Wed, 23 Oct 2019 03:12:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75C1A6B0008; Wed, 23 Oct 2019 03:12:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 673DC6B000A; Wed, 23 Oct 2019 03:12:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0209.hostedemail.com [216.40.44.209]) by kanga.kvack.org (Postfix) with ESMTP id 45BAE6B0007 for ; Wed, 23 Oct 2019 03:12:22 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id EF04B814F for ; Wed, 23 Oct 2019 07:12:21 +0000 (UTC) X-FDA: 76074180882.12.part03_6bdbdfa8ac861 X-HE-Tag: part03_6bdbdfa8ac861 X-Filterd-Recvd-Size: 6542 Received: from mail105.syd.optusnet.com.au (mail105.syd.optusnet.com.au [211.29.132.249]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Wed, 23 Oct 2019 07:12:21 +0000 (UTC) Received: from dread.disaster.area (pa49-180-40-48.pa.nsw.optusnet.com.au [49.180.40.48]) by mail105.syd.optusnet.com.au (Postfix) with ESMTPS id 2AA3C362383; Wed, 23 Oct 2019 18:12:19 +1100 (AEDT) Received: from dave by dread.disaster.area with local (Exim 4.92.3) (envelope-from ) id 1iNAob-0001M3-PF; Wed, 23 Oct 2019 18:12:17 +1100 Date: Wed, 23 Oct 2019 18:12:17 +1100 From: Dave Chinner To: "Darrick J. Wong" Cc: Andrew Morton , linux-mm@kvack.org, linux-xfs@vger.kernel.org, bugzilla-daemon@bugzilla.kernel.org, goodmirek@goodmirek.com, Hillf Danton , Dmitry Vyukov , Joel Fernandes , Tetsuo Handa Subject: Re: [Bug 205135] System hang up when memory swapping (kswapd deadlock) Message-ID: <20191023071217.GD2044@dread.disaster.area> References: <20191022152422.e47fda82879dc7cd1f3cf5e5@linux-foundation.org> <20191023012228.GP913374@magnolia> <20191023064905.GC2044@dread.disaster.area> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20191023064905.GC2044@dread.disaster.area> User-Agent: Mutt/1.10.1 (2018-07-13) X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=G6BsK5s5 c=1 sm=1 tr=0 a=y881pOMu+B+mZdf5UrsJdA==:117 a=y881pOMu+B+mZdf5UrsJdA==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=kj9zAlcOel0A:10 a=XobE76Q3jBoA:10 a=VwQbUJbxAAAA:8 a=58vdV8W1AAAA:8 a=QY18SFpNAAAA:8 a=vTr9H3xdAAAA:8 a=7-415B0cAAAA:8 a=bUm4aOFkq6hwphnUWp0A:9 a=cVwwTdr4mUQTMu6D:21 a=f4NPki9qiVyKbICw:21 a=CjuIK1q_8ugA:10 a=AjGcO6oz07-iQ99wixmX:22 a=LkGzKdMokiVGLp8DTWHT:22 a=LYL6_n6_bXSRrjLcjcND:22 a=7PCjnrUJ-F5voXmZD6jJ:22 a=biEYGPWJfzWAr4FL6Ov7:22 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Oct 23, 2019 at 05:49:05PM +1100, Dave Chinner wrote: > On Tue, Oct 22, 2019 at 06:22:28PM -0700, Darrick J. Wong wrote: > > On Tue, Oct 22, 2019 at 03:24:22PM -0700, Andrew Morton wrote: > > > > > > (switched to email. Please respond via emailed reply-to-all, not via the > > > bugzilla web interface). > > > > > > On Tue, 22 Oct 2019 09:02:22 +0000 bugzilla-daemon@bugzilla.kernel.org wrote: > > > > > > > https://bugzilla.kernel.org/show_bug.cgi?id=205135 > > > > > > > > --- Comment #7 from goodmirek@goodmirek.com --- > > > > Everyone who uses a swapfile on XFS filesystem seem affected by this hang up. > > > > Not sure about other filesystems, I did not have a chance to test it elsewhere. > > > > > > > > This unreproduced bot crash could be related: > > > > https://lore.kernel.org/linux-mm/20190910071804.2944-1-hdanton@sina.com/ > > > > > > Thanks. Might be core MM, might be XFS, might be Fedora. > > > > > > Hilf, does your patch look related? That seems to have gone quiet? > > > > > > Should we progress Tetsuo's patch? > > > > Hmm... > > > > Oct 09 15:44:52 kernel: Linux version 5.4.0-0.rc1.git1.1.fc32.x86_64 (mockbuild@bkernel03.phx2.fedoraproject.org) (gcc version 9.2.1 20190827 (Red Hat 9.2.1-1) (GCC)) #1 SMP Fri Oct 4 14:57:23 UTC 2019 > > > > ...istr 5.4-rc1 had some writeback bugs in it... > > > > -> #1 (fs_reclaim){+.+.}: > > Oct 09 13:47:08 kernel: fs_reclaim_acquire.part.0+0x25/0x30 > > Oct 09 13:47:08 kernel: __kmalloc+0x4f/0x330 > > Oct 09 13:47:08 kernel: kmem_alloc+0x83/0x1a0 [xfs] > > Oct 09 13:47:08 kernel: kmem_alloc_large+0x3c/0x100 [xfs] > > Oct 09 13:47:08 kernel: xfs_attr_copy_value+0x5d/0xa0 [xfs] > > Oct 09 13:47:08 kernel: xfs_attr_get+0xe7/0x1d0 [xfs] > > Oct 09 13:47:08 kernel: xfs_get_acl+0xad/0x1e0 [xfs] > > Oct 09 13:47:08 kernel: get_acl+0x81/0x110 > > Oct 09 13:47:08 kernel: posix_acl_create+0x58/0x160 > > Oct 09 13:47:08 kernel: xfs_generic_create+0x7e/0x2f0 [xfs] > > Oct 09 13:47:08 kernel: lookup_open+0x5bd/0x820 > > Oct 09 13:47:08 kernel: path_openat+0x340/0xcb0 > > Oct 09 13:47:08 kernel: do_filp_open+0x91/0x100 > > Oct 09 13:47:08 kernel: do_sys_open+0x184/0x220 > > Oct 09 13:47:08 kernel: do_syscall_64+0x5c/0xa0 > > Oct 09 13:47:08 kernel: entry_SYSCALL_64_after_hwframe+0x49/0xbe > > > > That's XFS trying to allocate memory to load an acl off disk, only it > > looks this thread does a MAYFAIL allocation. It's a GFP_FS (since we > > don't set KM_NOFS) allocation so we recurse into fs reclaim, and the > > ACL-getter has locked the inode (which is probably why lockdep > > triggers). I wonder if that's really a deadlock vs. just super-slow > > behavior, but otoh I don't think we're supposed to allow reclaim to jump > > into the filesystems when the fs has locks held. > > > > That kmem_alloc_large should probably be changed to KM_NOFS. Dave? > > I suspect it's a false positive, but without the rest of the lockdep > trace I don't have any context to determine if there is actually a > deadlock vector there. Ok, I've looked at the bz now, and the rest of the trace is kswapd locking an inode from the superblock shrinker. That means I'm pretty certain this is a false positive and has nothing to do with whatever hang is occuring on the user's machine. These: Oct 09 14:00:18 kernel: DMA-API: cacheline tracking ENOMEM, dma-debug disabled occur when a radix_tree_insert() call fails, but I don't see a radix_tree_preload() call anywhere around that code to ensure the radix tree insert has memory available before locks are taken and the insert is attempted. Ahhhh: static RADIX_TREE(dma_active_cacheline, GFP_NOWAIT); Seems like that is guaranteed to fail under mempry pressure as it won't allow memory reclaim to block waiting for progress to be made. Hence I see nothing in the bug to back up the assertions that "Everyone who uses a swapfile on XFS filesystem seem affected by this hang up." There's no evidence at all that even points the subsystem that has hung. sysrq-w, sysrq-l and sysrq-t output are the first things we need from that machine to see if/where it is actually hung... Cheers, Dave. -- Dave Chinner david@fromorbit.com