From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx170.postini.com [74.125.245.170]) by kanga.kvack.org (Postfix) with SMTP id 3B4766B0005 for ; Tue, 12 Feb 2013 22:11:18 -0500 (EST) Date: Wed, 13 Feb 2013 13:40:56 +1030 From: Jonathan Woithe Subject: OOM triggered with plenty of memory free Message-ID: <20130213031056.GA32135@marvin.atrad.com.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-linux-mm@kvack.org List-ID: To: linux-mm@kvack.org Cc: Jonathan Woithe Hi all Apologies for the length of this post. We have a number of Linux systems in the field controlling some instrumentation. They are backed by UPS and local generators so uptimes are often several hundred days. Recently we have noticed that after a few hundred days of uptime one of these systems started to trigger the OOM killer repeatedly even though "free" reports plenty of memory free. After speaking with a few people at linux.conf.au it was thought that an exhaustion of lowmem may be implicated and the suggestion was to ask here to see if anyone had ideas as to what the underlying cause might be. The output of "ps auxw" is practically unchanged from the time the machine boots. Some system specifications: - CPU: i7 860 at 2.8 GHz - Mainboard: Advantech AIMB-780 - RAM: 4 GB - Kernel: 2.6.35.11 SMP, 32 bit (kernel.org kernel, no patches applied) - Userspace: Slackware 13.1 - Ethernet card: Netgear GA311, using the r8169 kernel driver The machine is set up to boot during which time the data acquisition program is started. The system then runs indefinitely. Beyond the data acquisition program the machine does very little. The memory usage of all processes on the system (including the data acquisition program) as reported by "ps" remains unchanged for the lifetime of the processes. The data acquisition obtains its data via UDP packets producing a data flow rate into the ethernet card of the order of 20 MBytes/s. This flows for 55 seconds at a time with a 5 second break before the next flow starts. At the end of this email I have included a number of memory-related details taken from the system while it was in the fault condition. Unfortunately I did not get a dump of slabinfo because at the time I didn't know it existed. Being a production system there was pressure to restore functionality as soon as possible, which a reboot was observed to do. However, I did capture a slabinfo (and several other things) soon after the reboot and have been able to obtain these again today (after 106 days uptime) and include all these below as well. I have however been able to obtain the slabinfo from another machine operating in a similar way to the one which faulted. Its uptime is just over 100 days and has not yet started misbehaving. I have noted that the active_objs and num_objs fields for the kmalloc-32, kmalloc-64 and kmalloc-128 lines in particular are very large and seem to be increasing without limit over both short and long timescales (eg: over an hour, the kmalloc-128 active_objs increased from 2093227 to 2094021). If the acquisition is stopped (ie: software idling, hardware not sending data to ethernet port) these statistics do not appear to increase over a 30 minute period. I do not know whether this is significant. The kernel config applicable to the machine is at the end of this post. Suggesting to the operators of these systems that they need to reboot once every 100 days doesn't look good for anyone so I am keen to identify the problem. Our access to most of the affected systems is limited due to their operational status but I am willing to try to carry out any other tests you might suggest to narrow the problem down. I should note that moving these systems to a 64-bit kernel, while being a long term goal, is not something we can do in the shorter term. Regards jonathan The first OOM report (about 3 days before we were made aware of the problem): kernel: ftp invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0 kernel: Pid: 22217, comm: ftp Not tainted 2.6.35.11-smp #2 kernel: Call Trace: kernel: [] dump_header+0x65/0x1a0 kernel: [] ? ___ratelimit+0x82/0x100 kernel: [] oom_kill_process+0x5c/0x150 kernel: [] ? select_bad_process+0xa6/0xd0 kernel: [] __out_of_memory+0x51/0xb0 kernel: [] out_of_memory+0x57/0xd0 kernel: [] __alloc_pages_nodemask+0x5bc/0x5d0 kernel: [] tcp_sendmsg+0x683/0xa90 kernel: [] ? touch_atime+0xf1/0x130 kernel: [] sock_aio_write+0x128/0x140 kernel: [] ? xfs_file_aio_read+0x1c9/0x310 kernel: [] do_sync_write+0xa4/0xe0 kernel: [] ? security_file_permission+0x14/0x20 kernel: [] ? rw_verify_area+0x62/0xe0 kernel: [] ? do_sigaction+0x175/0x1b0 kernel: [] vfs_write+0x179/0x190 kernel: [] sys_write+0x42/0x70 kernel: [] syscall_call+0x7/0xb kernel: [] ? lpfc_pci_remove_one+0x24/0x87 kernel: Mem-Info: kernel: DMA per-cpu: kernel: CPU 0: hi: 0, btch: 1 usd: 0 kernel: CPU 1: hi: 0, btch: 1 usd: 0 kernel: CPU 2: hi: 0, btch: 1 usd: 0 kernel: CPU 3: hi: 0, btch: 1 usd: 0 kernel: CPU 4: hi: 0, btch: 1 usd: 0 kernel: CPU 5: hi: 0, btch: 1 usd: 0 kernel: CPU 6: hi: 0, btch: 1 usd: 0 kernel: CPU 7: hi: 0, btch: 1 usd: 0 kernel: Normal per-cpu: kernel: CPU 0: hi: 186, btch: 31 usd: 176 kernel: CPU 1: hi: 186, btch: 31 usd: 158 kernel: CPU 2: hi: 186, btch: 31 usd: 173 kernel: CPU 3: hi: 186, btch: 31 usd: 46 kernel: CPU 4: hi: 186, btch: 31 usd: 178 kernel: CPU 5: hi: 186, btch: 31 usd: 178 kernel: CPU 6: hi: 186, btch: 31 usd: 157 kernel: CPU 7: hi: 186, btch: 31 usd: 183 kernel: HighMem per-cpu: kernel: CPU 0: hi: 186, btch: 31 usd: 34 kernel: CPU 1: hi: 186, btch: 31 usd: 163 kernel: CPU 2: hi: 186, btch: 31 usd: 81 kernel: CPU 3: hi: 186, btch: 31 usd: 43 kernel: CPU 4: hi: 186, btch: 31 usd: 168 kernel: CPU 5: hi: 186, btch: 31 usd: 47 kernel: CPU 6: hi: 186, btch: 31 usd: 64 kernel: CPU 7: hi: 186, btch: 31 usd: 107 kernel: active_anon:11992 inactive_anon:5101 isolated_anon:0 kernel: active_file:6520 inactive_file:3463 isolated_file:32 kernel: unevictable:31758 dirty:1 writeback:3 unstable:0 kernel: free:488104 slab_reclaimable:1930 slab_unreclaimable:202381 kernel: mapped:5396 shmem:628 pagetables:378 bounce:0 kernel: DMA free:3480kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:52kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15800kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:148kB slab_unreclaimable:12148kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:151 all_unreclaimable? yes kernel: lowmem_reserve[]: 0 865 2991 2991 kernel: Normal free:3856kB min:3728kB low:4660kB high:5592kB active_anon:0kB inactive_anon:0kB active_file:916kB inactive_file:1068kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:7572kB slab_unreclaimable:797376kB kernel_stack:3136kB pagetables:36kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3616 all_unreclaimable? yes kernel: lowmem_reserve[]: 0 0 17014 17014 kernel: HighMem free:1945080kB min:512kB low:2800kB high:5092kB active_anon:47968kB inactive_anon:20404kB active_file:25152kB inactive_file:12732kB unevictable:127032kB isolated(anon):0kB isolated(file):0kB present:2177800kB mlocked:127032kB dirty:4kB writeback:12kB mapped:21584kB shmem:2512kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1476kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no kernel: lowmem_reserve[]: 0 0 0 0 kernel: DMA: 19*4kB 30*8kB 32*16kB 23*32kB 18*64kB 4*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3484kB kernel: Normal: 366*4kB 16*8kB 16*16kB 1*32kB 1*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3608kB kernel: HighMem: 963*4kB 4290*8kB 4922*16kB 4280*32kB 3257*64kB 2543*128kB 1554*256kB 847*512kB 272*1024kB 21*2048kB 1*4096kB = 1944956kB kernel: 11606 total pagecache pages kernel: 201 pages in swap cache kernel: Swap cache stats: add 3212, delete 3011, find 2174/2388 kernel: Free swap = 8392932kB kernel: Total swap = 8393924kB kernel: 776048 pages RAM kernel: 548738 pages HighMem kernel: 13801 pages reserved kernel: 373240 pages shared kernel: 263808 pages non-shared buddy_info in fault condition: Node 0, zone DMA 23 32 31 22 19 4 1 0 0 0 0 Node 0, zone Normal 510 9 13 8 5 2 3 0 1 0 0 Node 0, zone HighMem 280 204 82 366 2662 2167 1455 897 375 76 21 output from "free" in fault condition: total used free shared buffers cached Mem: 3048988 1101120 1947868 0 0 48780 -/+ buffers/cache: 1052340 1996648 Swap: 8393924 0 8393924 meminfo in fault condition: MemTotal: 3048988 kB MemFree: 1930548 kB Buffers: 0 kB Cached: 56876 kB SwapCached: 0 kB Active: 78016 kB Inactive: 53500 kB Active(anon): 57220 kB Inactive(anon): 22888 kB Active(file): 20796 kB Inactive(file): 30612 kB Unevictable: 127172 kB Mlocked: 127172 kB HighTotal: 2194952 kB HighFree: 1923040 kB LowTotal: 854036 kB LowFree: 7508 kB SwapTotal: 8393924 kB SwapFree: 8393924 kB Dirty: 52 kB Writeback: 684 kB AnonPages: 202204 kB Mapped: 25208 kB Shmem: 2600 kB Slab: 818868 kB SReclaimable: 6240 kB SUnreclaim: 812628 kB KernelStack: 2608 kB PageTables: 1388 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 9918416 kB Committed_AS: 433632 kB VmallocTotal: 122880 kB VmallocUsed: 24952 kB VmallocChunk: 56908 kB DirectMap4k: 16376 kB DirectMap4M: 892928 kB vmstat output in fault condition: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 1949852 0 48064 0 0 0 0 0 0 4 1 95 0 slabinfo soon after system reboot (all processes are running): slabinfo - version: 2.1 # name : tunables : slabdata UDPLITEv6 0 0 704 23 4 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 69 69 704 23 4 : tunables 0 0 0 : slabdata 3 3 0 tw_sock_TCPv6 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 TCPv6 115 115 1408 23 8 : tunables 0 0 0 : slabdata 5 5 0 nf_conntrack_c1e41520 408 408 240 34 2 : tunables 0 0 0 : slabdata 12 12 0 fuse_request 0 0 400 20 2 : tunables 0 0 0 : slabdata 0 0 0 fuse_inode 0 0 448 36 4 : tunables 0 0 0 : slabdata 0 0 0 dm_raid1_read_record 0 0 1056 31 8 : tunables 0 0 0 : slabdata 0 0 0 kcopyd_job 0 0 272 30 2 : tunables 0 0 0 : slabdata 0 0 0 dm_uevent 0 0 2464 13 8 : tunables 0 0 0 : slabdata 0 0 0 dm_rq_target_io 0 0 248 33 2 : tunables 0 0 0 : slabdata 0 0 0 i2o_block_req 36 36 2624 12 8 : tunables 0 0 0 : slabdata 3 3 0 kmalloc_dma-512 32 32 512 32 4 : tunables 0 0 0 : slabdata 1 1 0 kmalloc_dma-2048 32 32 2048 16 8 : tunables 0 0 0 : slabdata 2 2 0 fnic_sgl_max 0 0 16448 1 8 : tunables 0 0 0 : slabdata 0 0 0 fnic_sgl_dflt 0 0 576 28 4 : tunables 0 0 0 : slabdata 0 0 0 mqueue_inode_cache 28 28 576 28 4 : tunables 0 0 0 : slabdata 1 1 0 gfs2_quotad 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 gfs2_inode 0 0 528 31 4 : tunables 0 0 0 : slabdata 0 0 0 gfs2_glock(aspace) 0 0 352 23 2 : tunables 0 0 0 : slabdata 0 0 0 gfs2_glock 0 0 272 30 2 : tunables 0 0 0 : slabdata 0 0 0 extent_state 0 0 88 46 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_transaction_cache 0 0 144 28 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_inode_cache 0 0 632 25 4 : tunables 0 0 0 : slabdata 0 0 0 dlmfs_inode_cache 0 0 576 28 4 : tunables 0 0 0 : slabdata 0 0 0 ocfs2_dquot_cache 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0 ocfs2_inode_cache 0 0 1216 26 8 : tunables 0 0 0 : slabdata 0 0 0 xfs_ili 970 975 160 25 1 : tunables 0 0 0 : slabdata 39 39 0 xfs_inode 8899 8901 704 23 4 : tunables 0 0 0 : slabdata 387 387 0 xfs_efd_item 196 196 288 28 2 : tunables 0 0 0 : slabdata 7 7 0 xfs_buf_item 184 184 176 23 1 : tunables 0 0 0 : slabdata 8 8 0 xfs_trans 194 198 360 22 2 : tunables 0 0 0 : slabdata 9 9 0 xfs_da_state 168 168 336 24 2 : tunables 0 0 0 : slabdata 7 7 0 xfs_log_ticket 176 176 184 22 1 : tunables 0 0 0 : slabdata 8 8 0 jfs_mp 51 51 80 51 1 : tunables 0 0 0 : slabdata 1 1 0 jfs_ip 36 36 904 36 8 : tunables 0 0 0 : slabdata 1 1 0 udf_inode_cache 0 0 416 39 4 : tunables 0 0 0 : slabdata 0 0 0 romfs_i 0 0 368 22 2 : tunables 0 0 0 : slabdata 0 0 0 ntfs_big_inode_cache 0 0 576 28 4 : tunables 0 0 0 : slabdata 0 0 0 ntfs_inode_cache 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 nfsd4_delegations 0 0 240 34 2 : tunables 0 0 0 : slabdata 0 0 0 nfsd4_stateids 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 nfsd4_stateowners 0 0 344 23 2 : tunables 0 0 0 : slabdata 0 0 0 nfs_direct_cache 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 nfs_inode_cache 0 0 648 25 4 : tunables 0 0 0 : slabdata 0 0 0 isofs_inode_cache 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 fat_inode_cache 0 0 416 39 4 : tunables 0 0 0 : slabdata 0 0 0 fat_cache 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 jbd2_revoke_record 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 journal_head 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 revoke_record 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_inode_cache 0 0 632 25 4 : tunables 0 0 0 : slabdata 0 0 0 ext4_free_block_extents 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_alloc_context 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_prealloc_space 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_system_zone 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 ext2_inode_cache 0 0 504 32 4 : tunables 0 0 0 : slabdata 0 0 0 ext3_inode_cache 0 0 512 32 4 : tunables 0 0 0 : slabdata 0 0 0 ext3_xattr 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 reiser_inode_cache 0 0 448 36 4 : tunables 0 0 0 : slabdata 0 0 0 configfs_dir_cache 73 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 dquot 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 posix_timers_cache 34 34 120 34 1 : tunables 0 0 0 : slabdata 1 1 0 rpc_inode_cache 32 32 512 32 4 : tunables 0 0 0 : slabdata 1 1 0 UDP-Lite 0 0 576 28 4 : tunables 0 0 0 : slabdata 0 0 0 RAW 224 224 576 28 4 : tunables 0 0 0 : slabdata 8 8 0 UDP 196 196 576 28 4 : tunables 0 0 0 : slabdata 7 7 0 tw_sock_TCP 192 192 128 32 1 : tunables 0 0 0 : slabdata 6 6 0 TCP 175 175 1280 25 8 : tunables 0 0 0 : slabdata 7 7 0 sgpool-128 24 24 2560 12 8 : tunables 0 0 0 : slabdata 2 2 0 sgpool-64 25 25 1280 25 8 : tunables 0 0 0 : slabdata 1 1 0 sgpool-16 150 150 320 25 2 : tunables 0 0 0 : slabdata 6 6 0 blkdev_queue 110 110 1448 22 8 : tunables 0 0 0 : slabdata 5 5 0 blkdev_requests 288 288 224 36 2 : tunables 0 0 0 : slabdata 8 8 0 biovec-256 20 20 3072 10 8 : tunables 0 0 0 : slabdata 2 2 0 biovec-128 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 biovec-64 147 147 768 21 4 : tunables 0 0 0 : slabdata 7 7 0 sock_inode_cache 230 231 384 21 2 : tunables 0 0 0 : slabdata 11 11 0 skbuff_fclone_cache 168 168 384 21 2 : tunables 0 0 0 : slabdata 8 8 0 file_lock_cache 273 273 104 39 1 : tunables 0 0 0 : slabdata 7 7 0 net_namespace 0 0 1216 26 8 : tunables 0 0 0 : slabdata 0 0 0 shmem_inode_cache 1225 1225 456 35 4 : tunables 0 0 0 : slabdata 35 35 0 Acpi-Operand 3162 3162 40 102 1 : tunables 0 0 0 : slabdata 31 31 0 task_delay_info 589 612 80 51 1 : tunables 0 0 0 : slabdata 12 12 0 taskstats 24 24 328 24 2 : tunables 0 0 0 : slabdata 1 1 0 proc_inode_cache 1356 1407 384 21 2 : tunables 0 0 0 : slabdata 67 67 0 sigqueue 224 224 144 28 1 : tunables 0 0 0 : slabdata 8 8 0 bdev_cache 256 256 512 32 4 : tunables 0 0 0 : slabdata 8 8 0 sysfs_dir_cache 16056 16065 48 85 1 : tunables 0 0 0 : slabdata 189 189 0 inode_cache 5588 5588 360 22 2 : tunables 0 0 0 : slabdata 254 254 0 dentry 17532 17550 136 30 1 : tunables 0 0 0 : slabdata 585 585 0 buffer_head 4863 4964 56 73 1 : tunables 0 0 0 : slabdata 68 68 0 vm_area_struct 3120 3128 88 46 1 : tunables 0 0 0 : slabdata 68 68 0 mm_struct 356 360 448 36 4 : tunables 0 0 0 : slabdata 10 10 0 signal_cache 434 450 640 25 4 : tunables 0 0 0 : slabdata 18 18 0 sighand_cache 415 432 1344 24 8 : tunables 0 0 0 : slabdata 18 18 0 task_struct 501 522 1104 29 8 : tunables 0 0 0 : slabdata 18 18 0 anon_vma_chain 5513 5610 24 170 1 : tunables 0 0 0 : slabdata 33 33 0 anon_vma 3328 3328 16 256 1 : tunables 0 0 0 : slabdata 13 13 0 idr_layer_cache 520 520 152 26 1 : tunables 0 0 0 : slabdata 20 20 0 radix_tree_node 4427 4428 296 27 2 : tunables 0 0 0 : slabdata 164 164 0 kmalloc-8192 32 32 8192 4 8 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-4096 624 624 4096 8 8 : tunables 0 0 0 : slabdata 78 78 0 kmalloc-2048 409 416 2048 16 8 : tunables 0 0 0 : slabdata 26 26 0 kmalloc-1024 478 480 1024 32 8 : tunables 0 0 0 : slabdata 15 15 0 kmalloc-512 1003 1024 512 32 4 : tunables 0 0 0 : slabdata 32 32 0 kmalloc-256 346 352 256 32 2 : tunables 0 0 0 : slabdata 11 11 0 kmalloc-128 2217 3008 128 32 1 : tunables 0 0 0 : slabdata 94 94 0 kmalloc-64 4279 4288 64 64 1 : tunables 0 0 0 : slabdata 67 67 0 kmalloc-32 4984 4992 32 128 1 : tunables 0 0 0 : slabdata 39 39 0 kmalloc-16 6143 6144 16 256 1 : tunables 0 0 0 : slabdata 24 24 0 kmalloc-8 9213 9216 8 512 1 : tunables 0 0 0 : slabdata 18 18 0 kmalloc-192 1140 1218 192 21 1 : tunables 0 0 0 : slabdata 58 58 0 kmalloc-96 1806 1806 96 42 1 : tunables 0 0 0 : slabdata 43 43 0 slabinfo after 106 days uptime and continuous operation: slabinfo - version: 2.1 # name : tunables : slabdata UDPLITEv6 0 0 704 23 4 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 92 92 704 23 4 : tunables 0 0 0 : slabdata 4 4 0 tw_sock_TCPv6 63 63 192 21 1 : tunables 0 0 0 : slabdata 3 3 0 TCPv6 184 184 1408 23 8 : tunables 0 0 0 : slabdata 8 8 0 nf_conntrack_c1e41520 441 442 240 34 2 : tunables 0 0 0 : slabdata 13 13 0 fuse_request 0 0 400 20 2 : tunables 0 0 0 : slabdata 0 0 0 fuse_inode 0 0 448 36 4 : tunables 0 0 0 : slabdata 0 0 0 dm_raid1_read_record 0 0 1056 31 8 : tunables 0 0 0 : slabdata 0 0 0 kcopyd_job 0 0 272 30 2 : tunables 0 0 0 : slabdata 0 0 0 dm_uevent 0 0 2464 13 8 : tunables 0 0 0 : slabdata 0 0 0 dm_rq_target_io 0 0 248 33 2 : tunables 0 0 0 : slabdata 0 0 0 i2o_block_req 36 36 2624 12 8 : tunables 0 0 0 : slabdata 3 3 0 kmalloc_dma-512 32 32 512 32 4 : tunables 0 0 0 : slabdata 1 1 0 kmalloc_dma-2048 32 32 2048 16 8 : tunables 0 0 0 : slabdata 2 2 0 fnic_sgl_max 0 0 16448 1 8 : tunables 0 0 0 : slabdata 0 0 0 fnic_sgl_dflt 0 0 576 28 4 : tunables 0 0 0 : slabdata 0 0 0 mqueue_inode_cache 28 28 576 28 4 : tunables 0 0 0 : slabdata 1 1 0 gfs2_quotad 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 gfs2_inode 0 0 528 31 4 : tunables 0 0 0 : slabdata 0 0 0 gfs2_glock(aspace) 0 0 352 23 2 : tunables 0 0 0 : slabdata 0 0 0 gfs2_glock 0 0 272 30 2 : tunables 0 0 0 : slabdata 0 0 0 extent_state 0 0 88 46 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_transaction_cache 0 0 144 28 1 : tunables 0 0 0 : slabdata 0 0 0 btrfs_inode_cache 0 0 632 25 4 : tunables 0 0 0 : slabdata 0 0 0 dlmfs_inode_cache 0 0 576 28 4 : tunables 0 0 0 : slabdata 0 0 0 ocfs2_dquot_cache 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0 ocfs2_inode_cache 0 0 1216 26 8 : tunables 0 0 0 : slabdata 0 0 0 xfs_ili 571 700 160 25 1 : tunables 0 0 0 : slabdata 28 28 0 xfs_inode 28407 28474 704 23 4 : tunables 0 0 0 : slabdata 1238 1238 0 xfs_efd_item 224 224 288 28 2 : tunables 0 0 0 : slabdata 8 8 0 xfs_buf_item 184 184 176 23 1 : tunables 0 0 0 : slabdata 8 8 0 xfs_trans 183 198 360 22 2 : tunables 0 0 0 : slabdata 9 9 0 xfs_da_state 192 192 336 24 2 : tunables 0 0 0 : slabdata 8 8 0 xfs_log_ticket 176 176 184 22 1 : tunables 0 0 0 : slabdata 8 8 0 jfs_mp 51 51 80 51 1 : tunables 0 0 0 : slabdata 1 1 0 jfs_ip 36 36 904 36 8 : tunables 0 0 0 : slabdata 1 1 0 udf_inode_cache 0 0 416 39 4 : tunables 0 0 0 : slabdata 0 0 0 romfs_i 0 0 368 22 2 : tunables 0 0 0 : slabdata 0 0 0 ntfs_big_inode_cache 0 0 576 28 4 : tunables 0 0 0 : slabdata 0 0 0 ntfs_inode_cache 0 0 176 23 1 : tunables 0 0 0 : slabdata 0 0 0 nfsd4_delegations 0 0 240 34 2 : tunables 0 0 0 : slabdata 0 0 0 nfsd4_stateids 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 nfsd4_stateowners 0 0 344 23 2 : tunables 0 0 0 : slabdata 0 0 0 nfs_direct_cache 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 nfs_inode_cache 0 0 648 25 4 : tunables 0 0 0 : slabdata 0 0 0 isofs_inode_cache 0 0 384 21 2 : tunables 0 0 0 : slabdata 0 0 0 fat_inode_cache 0 0 416 39 4 : tunables 0 0 0 : slabdata 0 0 0 fat_cache 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 jbd2_revoke_record 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 journal_head 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 revoke_record 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_inode_cache 0 0 632 25 4 : tunables 0 0 0 : slabdata 0 0 0 ext4_free_block_extents 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_alloc_context 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_prealloc_space 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0 ext4_system_zone 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0 ext2_inode_cache 0 0 504 32 4 : tunables 0 0 0 : slabdata 0 0 0 ext3_inode_cache 0 0 512 32 4 : tunables 0 0 0 : slabdata 0 0 0 ext3_xattr 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0 reiser_inode_cache 0 0 448 36 4 : tunables 0 0 0 : slabdata 0 0 0 configfs_dir_cache 73 73 56 73 1 : tunables 0 0 0 : slabdata 1 1 0 dquot 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 posix_timers_cache 34 34 120 34 1 : tunables 0 0 0 : slabdata 1 1 0 rpc_inode_cache 32 32 512 32 4 : tunables 0 0 0 : slabdata 1 1 0 UDP-Lite 0 0 576 28 4 : tunables 0 0 0 : slabdata 0 0 0 RAW 246 252 576 28 4 : tunables 0 0 0 : slabdata 9 9 0 UDP 224 224 576 28 4 : tunables 0 0 0 : slabdata 8 8 0 tw_sock_TCP 256 256 128 32 1 : tunables 0 0 0 : slabdata 8 8 0 TCP 200 200 1280 25 8 : tunables 0 0 0 : slabdata 8 8 0 sgpool-128 24 24 2560 12 8 : tunables 0 0 0 : slabdata 2 2 0 sgpool-64 25 25 1280 25 8 : tunables 0 0 0 : slabdata 1 1 0 sgpool-16 200 200 320 25 2 : tunables 0 0 0 : slabdata 8 8 0 blkdev_queue 110 110 1448 22 8 : tunables 0 0 0 : slabdata 5 5 0 blkdev_requests 296 324 224 36 2 : tunables 0 0 0 : slabdata 9 9 0 biovec-256 20 20 3072 10 8 : tunables 0 0 0 : slabdata 2 2 0 biovec-128 0 0 1536 21 8 : tunables 0 0 0 : slabdata 0 0 0 biovec-64 179 189 768 21 4 : tunables 0 0 0 : slabdata 9 9 0 sock_inode_cache 238 252 384 21 2 : tunables 0 0 0 : slabdata 12 12 0 skbuff_fclone_cache 168 168 384 21 2 : tunables 0 0 0 : slabdata 8 8 0 file_lock_cache 312 312 104 39 1 : tunables 0 0 0 : slabdata 8 8 0 net_namespace 208 208 1216 26 8 : tunables 0 0 0 : slabdata 8 8 0 shmem_inode_cache 1224 1225 456 35 4 : tunables 0 0 0 : slabdata 35 35 0 Acpi-Operand 3366 3366 40 102 1 : tunables 0 0 0 : slabdata 33 33 0 task_delay_info 588 612 80 51 1 : tunables 0 0 0 : slabdata 12 12 0 taskstats 48 48 328 24 2 : tunables 0 0 0 : slabdata 2 2 0 proc_inode_cache 1262 1302 384 21 2 : tunables 0 0 0 : slabdata 62 62 0 sigqueue 224 224 144 28 1 : tunables 0 0 0 : slabdata 8 8 0 bdev_cache 256 256 512 32 4 : tunables 0 0 0 : slabdata 8 8 0 sysfs_dir_cache 16120 16235 48 85 1 : tunables 0 0 0 : slabdata 191 191 0 inode_cache 2520 2838 360 22 2 : tunables 0 0 0 : slabdata 129 129 0 dentry 38239 39000 136 30 1 : tunables 0 0 0 : slabdata 1300 1300 0 buffer_head 496273 640794 56 73 1 : tunables 0 0 0 : slabdata 8778 8778 0 vm_area_struct 3940 3956 88 46 1 : tunables 0 0 0 : slabdata 86 86 0 mm_struct 356 360 448 36 4 : tunables 0 0 0 : slabdata 10 10 0 signal_cache 433 450 640 25 4 : tunables 0 0 0 : slabdata 18 18 0 sighand_cache 429 432 1344 24 8 : tunables 0 0 0 : slabdata 18 18 0 task_struct 492 522 1104 29 8 : tunables 0 0 0 : slabdata 18 18 0 anon_vma_chain 7178 7310 24 170 1 : tunables 0 0 0 : slabdata 43 43 0 anon_vma 3824 3840 16 256 1 : tunables 0 0 0 : slabdata 15 15 0 idr_layer_cache 544 546 152 26 1 : tunables 0 0 0 : slabdata 21 21 0 radix_tree_node 20120 22329 296 27 2 : tunables 0 0 0 : slabdata 827 827 0 kmalloc-8192 32 32 8192 4 8 : tunables 0 0 0 : slabdata 8 8 0 kmalloc-4096 638 1016 4096 8 8 : tunables 0 0 0 : slabdata 127 127 0 kmalloc-2048 443 464 2048 16 8 : tunables 0 0 0 : slabdata 29 29 0 kmalloc-1024 507 544 1024 32 8 : tunables 0 0 0 : slabdata 17 17 0 kmalloc-512 1170 1216 512 32 4 : tunables 0 0 0 : slabdata 38 38 0 kmalloc-256 390 416 256 32 2 : tunables 0 0 0 : slabdata 13 13 0 kmalloc-128 1234556 1235168 128 32 1 : tunables 0 0 0 : slabdata 38599 38599 0 kmalloc-64 1238117 1238144 64 64 1 : tunables 0 0 0 : slabdata 19346 19346 0 kmalloc-32 1236600 1236608 32 128 1 : tunables 0 0 0 : slabdata 9661 9661 0 kmalloc-16 6143 6144 16 256 1 : tunables 0 0 0 : slabdata 24 24 0 kmalloc-8 9209 9216 8 512 1 : tunables 0 0 0 : slabdata 18 18 0 kmalloc-192 1814 1869 192 21 1 : tunables 0 0 0 : slabdata 89 89 0 kmalloc-96 2567 3066 96 42 1 : tunables 0 0 0 : slabdata 73 73 0 "free" output after 106 days runtime/operation: total used free shared buffers cached Mem: 3048988 2623536 425452 0 20 2070352 -/+ buffers/cache: 553164 2495824 Swap: 8393924 892 8393032 "ps auxw" output after 106 days runtime/operation: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 1.2 0.0 824 280 ? Ss 15:03 0:01 init [4] root 2 0.0 0.0 0 0 ? S 15:03 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 15:03 0:00 [ksoftirqd/0] root 4 0.0 0.0 0 0 ? S 15:03 0:00 [migration/0] root 5 0.0 0.0 0 0 ? S 15:03 0:00 [watchdog/0] root 6 0.0 0.0 0 0 ? S 15:03 0:00 [migration/1] root 7 0.0 0.0 0 0 ? S 15:03 0:00 [ksoftirqd/1] root 8 0.0 0.0 0 0 ? S 15:03 0:00 [watchdog/1] root 9 0.0 0.0 0 0 ? S 15:03 0:00 [migration/2] root 10 0.0 0.0 0 0 ? S 15:03 0:00 [ksoftirqd/2] root 11 0.0 0.0 0 0 ? S 15:03 0:00 [watchdog/2] root 12 0.0 0.0 0 0 ? S 15:03 0:00 [migration/3] root 13 0.0 0.0 0 0 ? S 15:03 0:00 [ksoftirqd/3] root 14 0.0 0.0 0 0 ? S 15:03 0:00 [watchdog/3] root 15 0.0 0.0 0 0 ? S 15:03 0:00 [migration/4] root 16 0.0 0.0 0 0 ? S 15:03 0:00 [ksoftirqd/4] root 17 0.0 0.0 0 0 ? S 15:03 0:00 [watchdog/4] root 18 0.0 0.0 0 0 ? S 15:03 0:00 [migration/5] root 19 0.0 0.0 0 0 ? S 15:03 0:00 [ksoftirqd/5] root 20 0.0 0.0 0 0 ? S 15:03 0:00 [watchdog/5] root 21 0.0 0.0 0 0 ? S 15:03 0:00 [migration/6] root 22 0.0 0.0 0 0 ? S 15:03 0:00 [ksoftirqd/6] root 23 0.0 0.0 0 0 ? S 15:03 0:00 [watchdog/6] root 24 0.0 0.0 0 0 ? S 15:03 0:00 [migration/7] root 25 0.0 0.0 0 0 ? S 15:03 0:00 [ksoftirqd/7] root 26 0.0 0.0 0 0 ? S 15:03 0:00 [watchdog/7] root 27 0.0 0.0 0 0 ? S 15:03 0:00 [events/0] root 28 0.0 0.0 0 0 ? S 15:03 0:00 [events/1] root 29 0.0 0.0 0 0 ? S 15:03 0:00 [events/2] root 30 0.2 0.0 0 0 ? S 15:03 0:00 [events/3] root 31 0.0 0.0 0 0 ? S 15:03 0:00 [events/4] root 32 0.0 0.0 0 0 ? S 15:03 0:00 [events/5] root 33 0.0 0.0 0 0 ? S 15:03 0:00 [events/6] root 34 0.0 0.0 0 0 ? S 15:03 0:00 [events/7] root 35 0.0 0.0 0 0 ? S 15:03 0:00 [cpuset] root 36 0.0 0.0 0 0 ? S 15:03 0:00 [khelper] root 39 0.0 0.0 0 0 ? S 15:03 0:00 [netns] root 42 0.0 0.0 0 0 ? S 15:03 0:00 [async/mgr] root 43 0.0 0.0 0 0 ? S 15:03 0:00 [pm] root 272 0.0 0.0 0 0 ? S 15:03 0:00 [sync_supers] root 274 0.0 0.0 0 0 ? S 15:03 0:00 [bdi-default] root 276 0.0 0.0 0 0 ? S 15:03 0:00 [kblockd/0] root 277 0.0 0.0 0 0 ? S 15:03 0:00 [kblockd/1] root 278 0.0 0.0 0 0 ? S 15:03 0:00 [kblockd/2] root 279 0.0 0.0 0 0 ? S 15:03 0:00 [kblockd/3] root 280 0.0 0.0 0 0 ? S 15:03 0:00 [kblockd/4] root 281 0.0 0.0 0 0 ? S 15:03 0:00 [kblockd/5] root 282 0.0 0.0 0 0 ? S 15:03 0:00 [kblockd/6] root 283 0.0 0.0 0 0 ? S 15:03 0:00 [kblockd/7] root 285 0.0 0.0 0 0 ? S 15:03 0:00 [kacpid] root 286 0.0 0.0 0 0 ? S 15:03 0:00 [kacpi_notify] root 287 0.0 0.0 0 0 ? S 15:03 0:00 [kacpi_hotplug] root 436 0.0 0.0 0 0 ? S 15:03 0:00 [ata_aux] root 437 0.0 0.0 0 0 ? S 15:03 0:00 [ata_sff/0] root 438 0.0 0.0 0 0 ? S 15:03 0:00 [ata_sff/1] root 439 0.0 0.0 0 0 ? S 15:03 0:00 [ata_sff/2] root 440 0.0 0.0 0 0 ? S 15:03 0:00 [ata_sff/3] root 441 0.0 0.0 0 0 ? S 15:03 0:00 [ata_sff/4] root 442 0.0 0.0 0 0 ? S 15:03 0:00 [ata_sff/5] root 443 0.0 0.0 0 0 ? S 15:03 0:00 [ata_sff/6] root 444 0.0 0.0 0 0 ? S 15:03 0:00 [ata_sff/7] root 449 0.0 0.0 0 0 ? S 15:03 0:00 [khubd] root 452 0.0 0.0 0 0 ? S 15:03 0:00 [kseriod] root 557 0.0 0.0 0 0 ? S 15:03 0:00 [rpciod/0] root 558 0.0 0.0 0 0 ? S 15:03 0:00 [rpciod/1] root 559 0.0 0.0 0 0 ? S 15:03 0:00 [rpciod/2] root 560 0.0 0.0 0 0 ? S 15:03 0:00 [rpciod/3] root 561 0.0 0.0 0 0 ? S 15:03 0:00 [rpciod/4] root 562 0.0 0.0 0 0 ? S 15:03 0:00 [rpciod/5] root 563 0.0 0.0 0 0 ? S 15:03 0:00 [rpciod/6] root 564 0.0 0.0 0 0 ? S 15:03 0:00 [rpciod/7] root 629 0.0 0.0 0 0 ? S 15:03 0:00 [khungtaskd] root 630 0.0 0.0 0 0 ? S 15:03 0:00 [kswapd0] root 683 0.0 0.0 0 0 ? S 15:03 0:00 [aio/0] root 684 0.0 0.0 0 0 ? S 15:03 0:00 [aio/1] root 685 0.0 0.0 0 0 ? S 15:03 0:00 [aio/2] root 686 0.0 0.0 0 0 ? S 15:03 0:00 [aio/3] root 687 0.0 0.0 0 0 ? S 15:03 0:00 [aio/4] root 688 0.0 0.0 0 0 ? S 15:03 0:00 [aio/5] root 689 0.0 0.0 0 0 ? S 15:03 0:00 [aio/6] root 690 0.0 0.0 0 0 ? S 15:03 0:00 [aio/7] root 711 0.0 0.0 0 0 ? S 15:03 0:00 [nfsiod] root 723 0.0 0.0 0 0 ? S 15:03 0:00 [jfsIO] root 724 0.0 0.0 0 0 ? S 15:03 0:00 [jfsCommit] root 725 0.0 0.0 0 0 ? S 15:03 0:00 [jfsCommit] root 726 0.0 0.0 0 0 ? S 15:03 0:00 [jfsCommit] root 727 0.0 0.0 0 0 ? S 15:03 0:00 [jfsCommit] root 728 0.0 0.0 0 0 ? S 15:03 0:00 [jfsCommit] root 729 0.0 0.0 0 0 ? S 15:03 0:00 [jfsCommit] root 730 0.0 0.0 0 0 ? S 15:03 0:00 [jfsCommit] root 731 0.0 0.0 0 0 ? S 15:03 0:00 [jfsCommit] root 732 0.0 0.0 0 0 ? S 15:03 0:00 [jfsSync] root 740 0.0 0.0 0 0 ? S 15:03 0:00 [xfs_mru_cache] root 741 0.0 0.0 0 0 ? S 15:03 0:00 [xfslogd/0] root 742 0.0 0.0 0 0 ? S 15:03 0:00 [xfslogd/1] root 743 0.0 0.0 0 0 ? S 15:03 0:00 [xfslogd/2] root 744 0.0 0.0 0 0 ? S 15:03 0:00 [xfslogd/3] root 745 0.0 0.0 0 0 ? S 15:03 0:00 [xfslogd/4] root 746 0.0 0.0 0 0 ? S 15:03 0:00 [xfslogd/5] root 747 0.0 0.0 0 0 ? S 15:03 0:00 [xfslogd/6] root 748 0.0 0.0 0 0 ? S 15:03 0:00 [xfslogd/7] root 749 0.0 0.0 0 0 ? S 15:03 0:00 [xfsdatad/0] root 750 0.0 0.0 0 0 ? S 15:03 0:00 [xfsdatad/1] root 751 0.0 0.0 0 0 ? S 15:03 0:00 [xfsdatad/2] root 752 0.0 0.0 0 0 ? S 15:03 0:00 [xfsdatad/3] root 753 0.0 0.0 0 0 ? S 15:03 0:00 [xfsdatad/4] root 754 0.0 0.0 0 0 ? S 15:03 0:00 [xfsdatad/5] root 755 0.0 0.0 0 0 ? S 15:03 0:00 [xfsdatad/6] root 756 0.0 0.0 0 0 ? S 15:03 0:00 [xfsdatad/7] root 757 0.0 0.0 0 0 ? S 15:03 0:00 [xfsconvertd/0] root 758 0.0 0.0 0 0 ? S 15:03 0:00 [xfsconvertd/1] root 759 0.0 0.0 0 0 ? S 15:03 0:00 [xfsconvertd/2] root 760 0.0 0.0 0 0 ? S 15:03 0:00 [xfsconvertd/3] root 761 0.0 0.0 0 0 ? S 15:03 0:00 [xfsconvertd/4] root 762 0.0 0.0 0 0 ? S 15:03 0:00 [xfsconvertd/5] root 763 0.0 0.0 0 0 ? S 15:03 0:00 [xfsconvertd/6] root 764 0.0 0.0 0 0 ? S 15:03 0:00 [xfsconvertd/7] root 767 0.0 0.0 0 0 ? S 15:03 0:00 [ocfs2_wq] root 768 0.0 0.0 0 0 ? S 15:03 0:00 [o2quot/0] root 769 0.0 0.0 0 0 ? S 15:03 0:00 [o2quot/1] root 770 0.0 0.0 0 0 ? S 15:03 0:00 [o2quot/2] root 771 0.0 0.0 0 0 ? S 15:03 0:00 [o2quot/3] root 772 0.0 0.0 0 0 ? S 15:03 0:00 [o2quot/4] root 773 0.0 0.0 0 0 ? S 15:03 0:00 [o2quot/5] root 774 0.0 0.0 0 0 ? S 15:03 0:00 [o2quot/6] root 775 0.0 0.0 0 0 ? S 15:03 0:00 [o2quot/7] root 778 0.0 0.0 0 0 ? S 15:03 0:00 [user_dlm] root 784 0.0 0.0 0 0 ? S 15:03 0:00 [glock_workqueue] root 785 0.0 0.0 0 0 ? S 15:03 0:00 [glock_workqueue] root 786 0.0 0.0 0 0 ? S 15:03 0:00 [glock_workqueue] root 787 0.0 0.0 0 0 ? S 15:03 0:00 [glock_workqueue] root 788 0.0 0.0 0 0 ? S 15:03 0:00 [glock_workqueue] root 789 0.0 0.0 0 0 ? S 15:03 0:00 [glock_workqueue] root 790 0.0 0.0 0 0 ? S 15:03 0:00 [glock_workqueue] root 791 0.0 0.0 0 0 ? S 15:03 0:00 [glock_workqueue] root 792 0.0 0.0 0 0 ? S 15:03 0:00 [delete_workqueu] root 793 0.0 0.0 0 0 ? S 15:03 0:00 [delete_workqueu] root 794 0.0 0.0 0 0 ? S 15:03 0:00 [delete_workqueu] root 795 0.0 0.0 0 0 ? S 15:03 0:00 [delete_workqueu] root 796 0.0 0.0 0 0 ? S 15:03 0:00 [delete_workqueu] root 797 0.0 0.0 0 0 ? S 15:03 0:00 [delete_workqueu] root 798 0.0 0.0 0 0 ? S 15:03 0:00 [delete_workqueu] root 799 0.0 0.0 0 0 ? S 15:03 0:00 [delete_workqueu] root 804 0.2 0.0 0 0 ? S< 15:03 0:00 [kslowd000] root 805 0.2 0.0 0 0 ? S< 15:03 0:00 [kslowd001] root 807 0.0 0.0 0 0 ? S 15:03 0:00 [crypto/0] root 808 0.0 0.0 0 0 ? S 15:03 0:00 [crypto/1] root 809 0.0 0.0 0 0 ? S 15:03 0:00 [crypto/2] root 810 0.0 0.0 0 0 ? S 15:03 0:00 [crypto/3] root 811 0.0 0.0 0 0 ? S 15:03 0:00 [crypto/4] root 812 0.0 0.0 0 0 ? S 15:03 0:00 [crypto/5] root 813 0.0 0.0 0 0 ? S 15:03 0:00 [crypto/6] root 814 0.0 0.0 0 0 ? S 15:03 0:00 [crypto/7] root 1011 0.0 0.0 0 0 ? S 15:03 0:00 [cciss_scan] root 1016 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_tgtd/0] root 1017 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_tgtd/1] root 1018 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_tgtd/2] root 1019 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_tgtd/3] root 1020 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_tgtd/4] root 1021 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_tgtd/5] root 1022 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_tgtd/6] root 1023 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_tgtd/7] root 1037 0.0 0.0 0 0 ? S 15:03 0:00 [iscsi_eh] root 1044 0.0 0.0 0 0 ? S 15:03 0:00 [fc_exch_workque] root 1045 0.0 0.0 0 0 ? S 15:03 0:00 [fc_rport_eq] root 1046 0.0 0.0 0 0 ? S< 15:03 0:00 [fcoethread/0] root 1047 0.0 0.0 0 0 ? S< 15:03 0:00 [fcoethread/1] root 1048 0.0 0.0 0 0 ? S< 15:03 0:00 [fcoethread/2] root 1049 0.0 0.0 0 0 ? S< 15:03 0:00 [fcoethread/3] root 1050 0.0 0.0 0 0 ? S< 15:03 0:00 [fcoethread/4] root 1051 0.0 0.0 0 0 ? S< 15:03 0:00 [fcoethread/5] root 1052 0.0 0.0 0 0 ? S< 15:03 0:00 [fcoethread/6] root 1053 0.0 0.0 0 0 ? S< 15:03 0:00 [fcoethread/7] root 1056 0.0 0.0 0 0 ? S 15:03 0:00 [fnic_event_wq] root 1116 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_eh_2] root 1139 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_eh_3] root 1142 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_eh_4] root 1145 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_eh_5] root 1148 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_eh_6] root 1151 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_eh_7] root 1154 0.0 0.0 0 0 ? S 15:03 0:00 [scsi_eh_8] root 1258 0.0 0.0 0 0 ? S 15:03 0:00 [exec-osm/0] root 1259 0.0 0.0 0 0 ? S 15:03 0:00 [exec-osm/1] root 1260 0.0 0.0 0 0 ? S 15:03 0:00 [exec-osm/2] root 1261 0.0 0.0 0 0 ? S 15:03 0:00 [exec-osm/3] root 1262 0.0 0.0 0 0 ? S 15:03 0:00 [exec-osm/4] root 1263 0.0 0.0 0 0 ? S 15:03 0:00 [exec-osm/5] root 1264 0.0 0.0 0 0 ? S 15:03 0:00 [exec-osm/6] root 1265 0.0 0.0 0 0 ? S 15:03 0:00 [exec-osm/7] root 1271 0.0 0.0 0 0 ? S 15:03 0:00 [block-osm/0] root 1272 0.0 0.0 0 0 ? S 15:03 0:00 [block-osm/1] root 1273 0.0 0.0 0 0 ? S 15:03 0:00 [block-osm/2] root 1274 0.0 0.0 0 0 ? S 15:03 0:00 [block-osm/3] root 1275 0.0 0.0 0 0 ? S 15:03 0:00 [block-osm/4] root 1276 0.0 0.0 0 0 ? S 15:03 0:00 [block-osm/5] root 1277 0.0 0.0 0 0 ? S 15:03 0:00 [block-osm/6] root 1278 0.0 0.0 0 0 ? S 15:03 0:00 [block-osm/7] root 1333 0.0 0.0 0 0 ? S 15:03 0:00 [kstriped] root 1336 0.0 0.0 0 0 ? S 15:03 0:00 [ksnapd] root 1343 0.0 0.0 0 0 ? S 15:03 0:00 [xfsbufd/sda1] root 1344 0.0 0.0 0 0 ? S 15:03 0:00 [xfsaild/sda1] root 1348 0.0 0.0 0 0 ? S 15:03 0:00 [xfssyncd/sda1] root 1382 0.0 0.0 2344 912 ? S email@kvack.org