From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx158.postini.com [74.125.245.158]) by kanga.kvack.org (Postfix) with SMTP id 8330B6B0072 for ; Fri, 26 Oct 2012 14:48:07 -0400 (EDT) Date: Fri, 26 Oct 2012 14:48:05 -0400 From: "J. Bruce Fields" Subject: Re: spinning in isolate_migratepages_range on busy nfs server Message-ID: <20121026184805.GB13094@fieldses.org> References: <20121025164722.GE6846@fieldses.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20121025164722.GE6846@fieldses.org> Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, bmarson@redhat.com On Thu, Oct 25, 2012 at 12:47:22PM -0400, bfields wrote: > We're seeing an nfs server on a 3.6-ish kernel lock up after running > specfs for a while. > > Looking at the logs, there are some hung task warnings showing nfsd > threads stuck on directory i_mutexes trying to do lookups. > > A sysrq-t dump showed there were also lots of threads holding those > i_mutexes while trying to allocate xfs inodes: > > nfsd R running task 0 6517 2 0x00000080 > ffff880f925074c0 0000000000000046 ffff880fe4718000 ffff880f92507fd8 > ffff880f92507fd8 ffff880f92507fd8 ffff880fd7920000 ffff880fe4718000 > 0000000000000000 ffff880f92506000 ffff88102ffd96c0 ffff88102ffd9b40 > Call Trace: > [] __cond_resched+0x2a/0x40 > [] _cond_resched+0x30/0x40 > [] isolate_migratepages_range+0xb2/0x550 > [] ? compact_checklock_irqsave.isra.17+0xe0/0xe0 > [] compact_zone+0x146/0x3f0 > [] compact_zone_order+0x82/0xc0 > [] try_to_compact_pages+0xe1/0x110 > [] __alloc_pages_direct_compact+0xaa/0x190 > [] __alloc_pages_nodemask+0x517/0x980 > [] ? __synchronize_srcu+0xf0/0x110 > [] alloc_pages_current+0xb0/0x120 > [] new_slab+0x265/0x310 > [] __slab_alloc+0x358/0x525 > [] ? kmem_zone_alloc+0x67/0xf0 [xfs] > [] ? up+0x32/0x50 > [] ? kmem_zone_alloc+0x67/0xf0 [xfs] > [] kmem_cache_alloc+0xff/0x130 > [] kmem_zone_alloc+0x67/0xf0 [xfs] > [] xfs_inode_alloc+0x29/0x270 [xfs] > [] xfs_iget+0x231/0x6c0 [xfs] > [] xfs_lookup+0xe7/0x110 [xfs] > [] xfs_vn_lookup+0x51/0x90 [xfs] > [] lookup_real+0x1d/0x60 > [] __lookup_hash+0x38/0x50 > [] lookup_one_len+0xd6/0x110 > [] nfsd_lookup_dentry+0x12b/0x4a0 [nfsd] > [] nfsd_lookup+0x79/0x140 [nfsd] > [] nfsd3_proc_lookup+0xef/0x1c0 [nfsd] > [] nfsd_dispatch+0xeb/0x230 [nfsd] > [] svc_process_common+0x328/0x6d0 [sunrpc] > [] svc_process+0x102/0x150 [sunrpc] > [] nfsd+0xb5/0x1a0 [nfsd] > [] ? nfsd_get_default_max_blksize+0x60/0x60 [nfsd] > [] kthread+0x93/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread_freezable_should_stop+0x70/0x70 > [] ? gs_change+0x13/0x13 > > And perf --call-graph also shows we're spending all our time in the same > place, spinning on a lock (zone->lru_lock, I assume): > > - 92.65% nfsd [kernel.kallsyms] [k] _raw_spin_lock_irqsave > - _raw_spin_lock_irqsave > - 99.86% isolate_migratepages_range > > Just grepping through logs, I ran across 2a1402aa04 "mm: compaction: > acquire the zone->lru_lock as late as possible", in v3.7-rc1, which > looks relevant: > > Richard Davies and Shaohua Li have both reported lock contention > problems in compaction on the zone and LRU locks as well as > significant amounts of time being spent in compaction. This > series aims to reduce lock contention and scanning rates to > reduce that CPU usage. Richard reported at > https://lkml.org/lkml/2012/9/21/91 that this series made a big > different to a problem he reported in August: > > http://marc.info/?l=kvm&m=134511507015614&w=2 > > So we're trying that. Is there anything else we should try? Confirmed, applying that to 3.6 seems to fix the problem. --b. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org