From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail143.messagelabs.com (mail143.messagelabs.com [216.82.254.35]) by kanga.kvack.org (Postfix) with SMTP id 3CD758D0039 for ; Fri, 28 Jan 2011 09:36:42 -0500 (EST) Date: Fri, 28 Jan 2011 09:36:37 -0500 (EST) From: CAI Qian Message-ID: <10770737.209952.1296225397411.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> In-Reply-To: <1939528112.209753.1296225228879.JavaMail.root@zmail06.collab.prod.int.phx2.redhat.com> Subject: Re: known oom issues on numa in -mm tree? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: David Rientjes Cc: linux-mm , Andrea Arcangeli List-ID: ----- Original Message ----- > ----- Original Message ----- > > On Fri, 28 Jan 2011, CAI Qian wrote: > > > > > I can still reproduce this similar failure on both AMD and Intel > > > NUMA > > > systems using the latest linus tree with the commit you mentioned. > > > Unfortunately, I can't get a clear sysrq/console output of it but > > > only > > > a part of it (screenshot attached). > > > > > > It at least very easy to reproduce it for me by running LTP oom01 > > > test > > > for both Magny-Cours and Nehalem-EX NUMA systems. > > > > > > > Are you sure this is the same issue? The picture you provided > > doesn't > > show the top of the stack so I don't know what it's doing, but the > > original report had this: > > > > oom02 R running task 0 2023 1969 0x00000088 > > 0000000000000282 ffff88041d219df0 ffff88041fbf8ef0 ffffffff81100800 > > ffff880418ab5b18 0000000000000282 ffffffff8100c9ee ffff880418ab5ba8 > > 0000000087654321 0000000000000000 ffff880000000000 0000000000000001 > > Call Trace: > > [] ? drain_local_pages+0x0/0x20 > > [] ? apic_timer_interrupt+0xe/0x20 > > [] ? smp_call_function_many+0x1b6/0x210 > > [] ? smp_call_function_many+0x192/0x210 > > [] ? drain_local_pages+0x0/0x20 > > [] ? smp_call_function+0x22/0x30 > > [] ? on_each_cpu+0x24/0x50 > > [] ? drain_all_pages+0x1c/0x20 > > [] ? __alloc_pages_nodemask+0x4e4/0x840 > > [] ? alloc_page_vma+0x89/0x140 > > [] ? handle_mm_fault+0x871/0xd80 > > [] ? schedule+0x3fd/0x980 > > [] ? apic_timer_interrupt+0xe/0x20 > > [] ? apic_timer_interrupt+0xe/0x20 > > [] ? do_page_fault+0x143/0x4b0 > > [] ? __switch_to+0x194/0x320 > > [] ? schedule+0x3fd/0x980 > > [] ? page_fault+0x25/0x30 > > > > and the reported symptom was kswapd running excessively. I'm pretty > > sure > > I fixed that with 2ff754fa8f41 (mm: clear pages_scanned only if > > draining a > > pcp adds pages to the buddy allocator). > > > > Absent the dmesg, it's going to be very difficult to diagnose an > > issue > > that isn't a panic. > Finally, have been able to get a sysrq-t output when oom01 is > allocating > memory while used/free swap remained unchanged. All kswapd stopped at > zone_reclaim. > > # free -m > total used free shared buffers cached > Mem: 48392 48050 341 0 26 31 > -/+ buffers/cache: 47993 399 > Swap: 50447 29560 20887 > > > oom01 R running task 0 14534 14249 0x00000088 > ffff88063fc17d58 0000000000000086 ffff88063fc17d20 000000020000000e > 0000000000014d40 ffffea0013cc72d8 ffffea0013cc7188 ffffea0013cc7188 > 0000000000000297 ffff88063fc17d20 ffff8806371a7788 0000000000000297 > Call Trace: > [] ? release_pages+0x24e/0x260 > [] ? cpumask_any_but+0x31/0x50 > [] ? flush_tlb_mm+0x42/0xa0 > [] ? __pagevec_release+0x26/0x40 > [] ? move_active_pages_to_lru+0x19f/0x1d0 > [] ? apic_timer_interrupt+0xe/0x20 > [] ? apic_timer_interrupt+0xe/0x20 > [] ? apic_timer_interrupt+0xe/0x20 > [] ? apic_timer_interrupt+0xe/0x20 > [] ? zone_watermark_ok+0x1f/0x30 > [] ? compaction_suitable+0x3c/0xc0 > [] ? shrink_zone+0x1a7/0x520 > [] ? shrink_zone+0x12d/0x520 > [] ? ktime_get_ts+0xb3/0xf0 > [] ? do_try_to_free_pages+0xbf/0x4a0 > [] ? try_to_free_pages+0x92/0x130 > [] ? __alloc_pages_nodemask+0x45f/0x850 > [] ? alloc_pages_vma+0x93/0x150 > [] ? do_huge_pmd_anonymous_page+0x13a/0x330 > [] ? handle_mm_fault+0x24d/0x320 > [] ? do_page_fault+0x143/0x4b0 > [] ? schedule+0x44e/0xa10 > [] ? page_fault+0x25/0x30 > > kswapd0 S ffff88022e28d000 0 275 2 0x00000000 > ffff88022e2edda0 0000000000000046 ffffffff81e642c0 ffffffff00000000 > 0000000000014d40 ffff88022e28ca70 ffff88022e28d000 ffff88022e2edfd8 > ffff88022e28d008 0000000000014d40 ffff88022e2ec010 0000000000014d40 > Call Trace: > [] ? zone_reclaim+0x380/0x400 > [] kswapd+0xb16/0xc10 > [] ? kswapd+0x0/0xc10 > [] ? schedule+0x44e/0xa10 > [] ? autoremove_wake_function+0x0/0x40 > [] ? kswapd+0x0/0xc10 > [] kthread+0x96/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0xa0 > [] ? kernel_thread_helper+0x0/0x10 > kswapd1 S ffff88022e28da30 0 276 2 0x00000000 > ffff88022e2efda0 0000000000000046 ffff880337174000 ffff880300000000 > 0000000000014d40 ffff88022e28d4a0 ffff88022e28da30 ffff88022e2effd8 > ffff88022e28da38 0000000000014d40 ffff88022e2ee010 0000000000014d40 > Call Trace: > [] ? zone_reclaim+0x380/0x400 > [] kswapd+0xb16/0xc10 > [] ? kswapd+0x0/0xc10 > [] ? schedule+0x44e/0xa10 > [] ? autoremove_wake_function+0x0/0x40 > [] ? kswapd+0x0/0xc10 > [] kthread+0x96/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0xa0 > [] ? kernel_thread_helper+0x0/0x10 > kswapd2 S ffff88022e282690 0 277 2 0x00000000 > ffff88022e2d9da0 0000000000000046 ffff88052f904000 ffff880500000000 > 0000000000014d40 ffff88022e282100 ffff88022e282690 ffff88022e2d9fd8 > ffff88022e282698 0000000000014d40 ffff88022e2d8010 0000000000014d40 > Call Trace: > [] ? zone_reclaim+0x380/0x400 > [] kswapd+0xb16/0xc10 > [] ? kswapd+0x0/0xc10 > [] ? schedule+0x44e/0xa10 > [] ? autoremove_wake_function+0x0/0x40 > [] ? kswapd+0x0/0xc10 > [] kthread+0x96/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0xa0 > [] ? kernel_thread_helper+0x0/0x10 > kswapd3 S ffff88022e2830c0 0 278 2 0x00000000 > ffff88022e2dbda0 0000000000000046 ffff880637124000 ffff880600000000 > 0000000000014d40 ffff88022e282b30 ffff88022e2830c0 ffff88022e2dbfd8 > ffff88022e2830c8 0000000000014d40 ffff88022e2da010 0000000000014d40 > Call Trace: > [] ? zone_reclaim+0x380/0x400 > [] kswapd+0xb16/0xc10 > [] ? kswapd+0x0/0xc10 > [] ? schedule+0x44e/0xa10 > [] ? autoremove_wake_function+0x0/0x40 > [] ? kswapd+0x0/0xc10 > [] kthread+0x96/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0xa0 > [] ? kernel_thread_helper+0x0/0x10 > kswapd4 S ffff88022e283af0 0 279 2 0x00000000 > ffff88022e301da0 0000000000000046 ffff88082f908000 ffff880800000000 > 0000000000014d40 ffff88022e283560 ffff88022e283af0 ffff88022e301fd8 > ffff88022e283af8 0000000000014d40 ffff88022e300010 0000000000014d40 > Call Trace: > [] ? zone_reclaim+0x380/0x400 > [] kswapd+0xb16/0xc10 > [] ? kswapd+0x0/0xc10 > [] ? schedule+0x44e/0xa10 > [] ? autoremove_wake_function+0x0/0x40 > [] ? kswapd+0x0/0xc10 > [] kthread+0x96/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0xa0 > [] ? kernel_thread_helper+0x0/0x10 > kswapd5 S ffff88022e278650 0 280 2 0x00000000 > ffff88022e303da0 0000000000000046 ffff880937134000 ffff880900000000 > 0000000000014d40 ffff88022e2780c0 ffff88022e278650 ffff88022e303fd8 > ffff88022e278658 0000000000014d40 ffff88022e302010 0000000000014d40 > Call Trace: > [] ? zone_reclaim+0x380/0x400 > [] kswapd+0xb16/0xc10 > [] ? kswapd+0x0/0xc10 > [] ? schedule+0x44e/0xa10 > [] ? autoremove_wake_function+0x0/0x40 > [] ? kswapd+0x0/0xc10 > [] kthread+0x96/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0xa0 > [] ? kernel_thread_helper+0x0/0x10 > kswapd6 S ffff88022e279080 0 281 2 0x00000000 > ffff88022e2c5da0 0000000000000046 ffff880a37134000 ffff880a00000000 > 0000000000014d40 ffff88022e278af0 ffff88022e279080 ffff88022e2c5fd8 > ffff88022e279088 0000000000014d40 ffff88022e2c4010 0000000000014d40 > Call Trace: > [] ? zone_reclaim+0x380/0x400 > [] kswapd+0xb16/0xc10 > [] ? kswapd+0x0/0xc10 > [] ? schedule+0x44e/0xa10 > [] ? autoremove_wake_function+0x0/0x40 > [] ? kswapd+0x0/0xc10 > [] kthread+0x96/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0xa0 > [] ? kernel_thread_helper+0x0/0x10 > kswapd7 S ffff88022e279ab0 0 282 2 0x00000000 > ffff88022e289da0 0000000000000046 ffff880c2f934000 ffff880c00000000 > 0000000000014d40 ffff88022e279520 ffff88022e279ab0 ffff88022e289fd8 > ffff88022e279ab8 0000000000014d40 ffff88022e288010 0000000000014d40 > Call Trace: > [] ? zone_reclaim+0x380/0x400 > [] kswapd+0xb16/0xc10 > [] ? kswapd+0x0/0xc10 > [] ? schedule+0x44e/0xa10 > [] ? autoremove_wake_function+0x0/0x40 > [] ? kswapd+0x0/0xc10 > [] kthread+0x96/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0xa0 > [] ? kernel_thread_helper+0x0/0x10 > > migration/25 R running task 0 90 2 0x00000000 > ffff88022f1d9de0 0000000000000046 ffff88083fc54d40 0000000000000000 > 0000000000014d40 ffff88022f1d2ab0 ffff88022f1d3040 ffff88022f1d9fd8 > ffff88022f1d3048 0000000000014d40 ffff88022f1d8010 0000000000014d40 > Call Trace: > [] ? stop_machine_cpu_stop+0x0/0xe0 > [] cpu_stopper_thread+0x13d/0x1d0 > [] ? cpu_stopper_thread+0x0/0x1d0 > [] ? cpu_stopper_thread+0x0/0x1d0 > [] kthread+0x96/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0xa0 > [] ? kernel_thread_helper+0x0/0x10 > > kworker/25:0 S ffff88022f1d2610 0 91 2 0x00000000 > ffff88022f1fbe50 0000000000000046 ffff88063fc112c8 0000000000000082 > 0000000000014d40 ffff88022f1d2080 ffff88022f1d2610 ffff88022f1fbfd8 > ffff88022f1d2618 0000000000014d40 ffff88022f1fa010 0000000000014d40 > Call Trace: > [] worker_thread+0x261/0x3c0 > [] ? worker_thread+0x0/0x3c0 > [] kthread+0x96/0xa0 > [] kernel_thread_helper+0x4/0x10 > [] ? kthread+0x0/0xa0 > [] ? kernel_thread_helper+0x0/0x10 > > ... > > runnable tasks: > task PID tree-key switches prio exec-runtime sum-exec sum-sleep > ---------------------------------------------------------------------------------------------------------- > migration/25 90 2150.833956 29 0 2150.833956 0.000920 0.000000 / > R oom01 14534 582482.371064 96026 120 582482.371064 519502.918710 > 91644.125770 / > kworker/25:2 14592 582470.371064 4811 120 582470.371064 108.496370 > 207368.125892 / Then, hung tasks killed off. INFO: task jbd2/dm-0-8:618 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. jbd2/dm-0-8 D ffff88022c5dfa30 0 618 2 0x00000000 ffff88022c601b30 0000000000000046 ffff88022c601ae0 ffffffff00000000 0000000000014d40 ffff88022c5df4a0 ffff88022c5dfa30 ffff88022c601fd8 ffff88022c5dfa38 0000000000014d40 ffff88022c600010 0000000000014d40 Call Trace: [] ? sync_page+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_page+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] wait_on_page_bit+0x73/0x80 [] ? wake_bit_function+0x0/0x50 [] ? pagevec_lookup_tag+0x25/0x40 [] filemap_fdatawait_range+0x113/0x1a0 [] ? submit_bio+0x86/0x110 [] ? bio_alloc_bioset+0x5b/0xf0 [] filemap_fdatawait+0x2b/0x30 [] jbd2_journal_commit_transaction+0x74c/0x13f0 [jbd2] [] ? try_to_del_timer_sync+0x83/0xe0 [] kjournald2+0xb8/0x220 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] ? kjournald2+0x0/0x220 [jbd2] [] kthread+0x96/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0xa0 [] ? kernel_thread_helper+0x0/0x10 INFO: task flush-253:0:1626 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-253:0 D ffff88022e34c5d0 0 1626 2 0x00000080 ffff88022e873630 0000000000000046 ffff880337397c80 ffff880300000000 0000000000014d40 ffff88022e34c040 ffff88022e34c5d0 ffff88022e873fd8 ffff88022e34c5d8 0000000000014d40 ffff88022e872010 0000000000014d40 Call Trace: [] ? sync_buffer+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_buffer+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] ? sync_buffer+0x0/0x50 [] out_of_line_wait_on_bit+0x78/0x90 [] ? wake_bit_function+0x0/0x50 [] __wait_on_buffer+0x2e/0x30 [] bh_submit_read+0x68/0x90 [] ext4_read_block_bitmap+0x176/0x390 [ext4] [] ext4_mb_mark_diskspace_used+0x56/0x320 [ext4] [] ? ext4_mb_new_group_pa+0x1ed/0x2d0 [ext4] [] ext4_mb_new_blocks+0x249/0x500 [ext4] [] ? ext4_ext_find_extent+0x12d/0x320 [ext4] [] ? end_buffer_async_write+0x0/0x1a0 [] ext4_ext_map_blocks+0x58e/0x1f40 [ext4] [] ? ____cache_alloc_node+0x100/0x150 [] ? radix_tree_gang_lookup_tag_slot+0x95/0xe0 [] ? pagevec_lookup_tag+0x25/0x40 [] ext4_map_blocks+0xf4/0x210 [ext4] [] mpage_da_map_and_submit+0xb9/0x440 [ext4] [] ? jbd2_journal_start+0x13/0x20 [jbd2] [] ? ext4_journal_start_sb+0xf0/0x130 [ext4] [] ext4_da_writepages+0x2f5/0x640 [ext4] [] do_writepages+0x21/0x40 [] writeback_single_inode+0x98/0x240 [] writeback_sb_inodes+0xce/0x170 [] writeback_inodes_wb+0x99/0x160 [] wb_writeback+0x2bb/0x430 [] wb_do_writeback+0x22c/0x280 [] bdi_writeback_thread+0xb2/0x260 [] ? bdi_writeback_thread+0x0/0x260 [] ? bdi_writeback_thread+0x0/0x260 [] kthread+0x96/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0xa0 [] ? kernel_thread_helper+0x0/0x10 INFO: task master:2115 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. master D ffff880335951000 0 2115 1 0x00000080 ffff8803359e1a88 0000000000000082 ffff8803359e1cb4 0000000000000000 0000000000014d40 ffff880335950a70 ffff880335951000 ffff8803359e1fd8 ffff880335951008 0000000000014d40 ffff8803359e0010 0000000000014d40 Call Trace: [] do_get_write_access+0x29d/0x510 [jbd2] [] ? wake_bit_function+0x0/0x50 [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] [] __ext4_journal_get_write_access+0x3e/0x80 [ext4] [] ext4_reserve_inode_write+0x78/0xa0 [ext4] [] ? jbd2__journal_start+0xbb/0x100 [jbd2] [] ? ext4_dirty_inode+0x3d/0x60 [ext4] [] ext4_mark_inode_dirty+0x5d/0x230 [ext4] [] ? jbd2_journal_start+0x13/0x20 [jbd2] [] ? ext4_journal_start_sb+0xf0/0x130 [ext4] [] ext4_dirty_inode+0x3d/0x60 [ext4] [] __mark_inode_dirty+0x38/0x1e0 [] file_update_time+0xec/0x170 [] pipe_write+0x32f/0x680 [] do_sync_write+0xda/0x120 [] ? selinux_file_permission+0xf3/0x150 [] ? security_file_permission+0x23/0x90 [] vfs_write+0xc8/0x190 [] sys_write+0x51/0x90 [] system_call_fastpath+0x16/0x1b INFO: task pickup:2124 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. pickup D ffff880635fd25d0 0 2124 2115 0x00000080 ffff880635feda98 0000000000000086 0000000000000006 0000000000000000 0000000000014d40 ffff880635fd2040 ffff880635fd25d0 ffff880635fedfd8 ffff880635fd25d8 0000000000014d40 ffff880635fec010 0000000000014d40 Call Trace: [] do_get_write_access+0x29d/0x510 [jbd2] [] ? wake_bit_function+0x0/0x50 [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] [] __ext4_journal_get_write_access+0x3e/0x80 [ext4] [] ext4_reserve_inode_write+0x78/0xa0 [ext4] [] ? jbd2__journal_start+0xbb/0x100 [jbd2] [] ? ext4_dirty_inode+0x3d/0x60 [ext4] [] ext4_mark_inode_dirty+0x5d/0x230 [ext4] [] ? jbd2_journal_start+0x13/0x20 [jbd2] [] ? ext4_journal_start_sb+0xf0/0x130 [ext4] [] ext4_dirty_inode+0x3d/0x60 [ext4] [] __mark_inode_dirty+0x38/0x1e0 [] touch_atime+0x12b/0x170 [] pipe_read+0x2b4/0x4c0 [] do_sync_read+0xda/0x120 [] ? security_file_permission+0x8b/0x90 [] vfs_read+0xc5/0x190 [] sys_read+0x51/0x90 [] system_call_fastpath+0x16/0x1b INFO: task sadc:14685 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. sadc D ffff88022b929af0 0 14685 14683 0x00000084 ffff88022ae73e08 0000000000000086 0000000000000001 0000000000000000 0000000000014d40 ffff88022b929560 ffff88022b929af0 ffff88022ae73fd8 ffff88022b929af8 0000000000014d40 ffff88022ae72010 0000000000014d40 Call Trace: [] jbd2_log_wait_commit+0xbd/0x130 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] ext4_sync_file+0x12d/0x330 [ext4] [] vfs_fsync_range+0x79/0x90 [] vfs_fsync+0x1c/0x20 [] do_fsync+0x3a/0x60 [] sys_fdatasync+0x13/0x20 [] system_call_fastpath+0x16/0x1b INFO: task jbd2/dm-0-8:618 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. jbd2/dm-0-8 D ffff88022c5dfa30 0 618 2 0x00000000 ffff88022c601b30 0000000000000046 ffff88022c601ae0 ffffffff00000000 0000000000014d40 ffff88022c5df4a0 ffff88022c5dfa30 ffff88022c601fd8 ffff88022c5dfa38 0000000000014d40 ffff88022c600010 0000000000014d40 Call Trace: [] ? sync_page+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_page+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] wait_on_page_bit+0x73/0x80 [] ? wake_bit_function+0x0/0x50 [] ? pagevec_lookup_tag+0x25/0x40 [] filemap_fdatawait_range+0x113/0x1a0 [] ? submit_bio+0x86/0x110 [] ? bio_alloc_bioset+0x5b/0xf0 [] filemap_fdatawait+0x2b/0x30 [] jbd2_journal_commit_transaction+0x74c/0x13f0 [jbd2] [] ? try_to_del_timer_sync+0x83/0xe0 [] kjournald2+0xb8/0x220 [jbd2] [] ? autoremove_wake_function+0x0/0x40 [] ? kjournald2+0x0/0x220 [jbd2] [] kthread+0x96/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0xa0 [] ? kernel_thread_helper+0x0/0x10 INFO: task flush-253:0:1626 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-253:0 D ffff88022e34c5d0 0 1626 2 0x00000080 ffff88022e873630 0000000000000046 ffff880337397c80 ffff880300000000 0000000000014d40 ffff88022e34c040 ffff88022e34c5d0 ffff88022e873fd8 ffff88022e34c5d8 0000000000014d40 ffff88022e872010 0000000000014d40 Call Trace: [] ? sync_buffer+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_buffer+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] ? sync_buffer+0x0/0x50 [] out_of_line_wait_on_bit+0x78/0x90 [] ? wake_bit_function+0x0/0x50 [] __wait_on_buffer+0x2e/0x30 [] bh_submit_read+0x68/0x90 [] ext4_read_block_bitmap+0x176/0x390 [ext4] [] ext4_mb_mark_diskspace_used+0x56/0x320 [ext4] [] ? ext4_mb_new_group_pa+0x1ed/0x2d0 [ext4] [] ext4_mb_new_blocks+0x249/0x500 [ext4] [] ? ext4_ext_find_extent+0x12d/0x320 [ext4] [] ? end_buffer_async_write+0x0/0x1a0 [] ext4_ext_map_blocks+0x58e/0x1f40 [ext4] [] ? ____cache_alloc_node+0x100/0x150 [] ? radix_tree_gang_lookup_tag_slot+0x95/0xe0 [] ? pagevec_lookup_tag+0x25/0x40 [] ext4_map_blocks+0xf4/0x210 [ext4] [] mpage_da_map_and_submit+0xb9/0x440 [ext4] [] ? jbd2_journal_start+0x13/0x20 [jbd2] [] ? ext4_journal_start_sb+0xf0/0x130 [ext4] [] ext4_da_writepages+0x2f5/0x640 [ext4] [] do_writepages+0x21/0x40 [] writeback_single_inode+0x98/0x240 [] writeback_sb_inodes+0xce/0x170 [] writeback_inodes_wb+0x99/0x160 [] wb_writeback+0x2bb/0x430 [] wb_do_writeback+0x22c/0x280 [] bdi_writeback_thread+0xb2/0x260 [] ? bdi_writeback_thread+0x0/0x260 [] ? bdi_writeback_thread+0x0/0x260 [] kthread+0x96/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? kthread+0x0/0xa0 [] ? kernel_thread_helper+0x0/0x10 INFO: task ntpd:2039 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ntpd D ffff88093661f080 0 2039 1 0x00000080 ffff880935f33ab8 0000000000000082 ffff880337397c80 ffff880335e344f8 0000000000014d40 ffff88093661eaf0 ffff88093661f080 ffff880935f33fd8 ffff88093661f088 0000000000014d40 ffff880935f32010 0000000000014d40 Call Trace: [] ? sync_buffer+0x0/0x50 [] io_schedule+0x70/0xc0 [] sync_buffer+0x40/0x50 [] __wait_on_bit+0x5f/0x90 [] ? bio_alloc_bioset+0x5b/0xf0 [] ? sync_buffer+0x0/0x50 [] out_of_line_wait_on_bit+0x78/0x90 [] ? wake_bit_function+0x0/0x50 [] __wait_on_buffer+0x2e/0x30 [] ext4_find_entry+0x3e8/0x4b0 [ext4] [] ? inode_has_perm+0x51/0xa0 [] ? ext4_lookup+0xb8/0x130 [ext4] [] ext4_lookup+0x4d/0x130 [ext4] [] d_alloc_and_lookup+0x45/0x90 [] ? d_lookup+0x35/0x60 [] __lookup_hash+0x106/0x1c0 [] do_last+0x14d/0x380 [] do_filp_open+0x3d8/0x770 [] ? sys_adjtimex+0x78/0x90 [] ? alloc_fd+0x95/0x160 [] do_sys_open+0x69/0x110 [] sys_open+0x20/0x30 [] system_call_fastpath+0x16/0x1b INFO: task master:2115 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. master D ffff880335951000 0 2115 1 0x00000084 ffff8803359e1a88 0000000000000082 ffff8803359e1cb4 0000000000000000 0000000000014d40 ffff880335950a70 ffff880335951000 ffff8803359e1fd8 ffff880335951008 0000000000014d40 ffff8803359e0010 0000000000014d40 Call Trace: [] do_get_write_access+0x29d/0x510 [jbd2] [] ? wake_bit_function+0x0/0x50 [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] [] __ext4_journal_get_write_access+0x3e/0x80 [ext4] [] ext4_reserve_inode_write+0x78/0xa0 [ext4] [] ? jbd2__journal_start+0xbb/0x100 [jbd2] [] ? ext4_dirty_inode+0x3d/0x60 [ext4] [] ext4_mark_inode_dirty+0x5d/0x230 [ext4] [] ? jbd2_journal_start+0x13/0x20 [jbd2] [] ? ext4_journal_start_sb+0xf0/0x130 [ext4] [] ext4_dirty_inode+0x3d/0x60 [ext4] [] __mark_inode_dirty+0x38/0x1e0 [] file_update_time+0xec/0x170 [] pipe_write+0x32f/0x680 [] do_sync_write+0xda/0x120 [] ? selinux_file_permission+0xf3/0x150 [] ? security_file_permission+0x23/0x90 [] vfs_write+0xc8/0x190 [] sys_write+0x51/0x90 [] system_call_fastpath+0x16/0x1b INFO: task pickup:2124 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. pickup D ffff880635fd25d0 0 2124 2115 0x00000080 ffff880635feda98 0000000000000086 0000000000000006 0000000000000000 0000000000014d40 ffff880635fd2040 ffff880635fd25d0 ffff880635fedfd8 ffff880635fd25d8 0000000000014d40 ffff880635fec010 0000000000014d40 Call Trace: [] do_get_write_access+0x29d/0x510 [jbd2] [] ? wake_bit_function+0x0/0x50 [] jbd2_journal_get_write_access+0x31/0x50 [jbd2] [] __ext4_journal_get_write_access+0x3e/0x80 [ext4] [] ext4_reserve_inode_write+0x78/0xa0 [ext4] [] ? jbd2__journal_start+0xbb/0x100 [jbd2] [] ? ext4_dirty_inode+0x3d/0x60 [ext4] [] ext4_mark_inode_dirty+0x5d/0x230 [ext4] [] ? jbd2_journal_start+0x13/0x20 [jbd2] [] ? ext4_journal_start_sb+0xf0/0x130 [ext4] [] ext4_dirty_inode+0x3d/0x60 [ext4] [] __mark_inode_dirty+0x38/0x1e0 [] touch_atime+0x12b/0x170 [] pipe_read+0x2b4/0x4c0 [] do_sync_read+0xda/0x120 [] ? security_file_permission+0x8b/0x90 [] vfs_read+0xc5/0x190 [] sys_read+0x51/0x90 [] system_call_fastpath+0x16/0x1b -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: email@kvack.org