kdb> ps Task Addr Pid Parent [*] cpu State Thread Command 0xc1e3c000 00000001 00000000 0 000 stop 0xc1e3c260 init 0xc7f9c000 00000002 00000001 0 000 run 0xc7f9c260 kapmd 0xc7f9a000 00000003 00000001 0 000 run 0xc7f9a260 kswapd 0xc7f98000 00000004 00000001 0 000 run 0xc7f98260 kreclaimd 0xc7f96000 00000005 00000001 0 000 run 0xc7f96260 kflushd 0xc7f94000 00000006 00000001 0 000 run 0xc7f94260 kupdate 0xc3e1e000 00000082 00000001 0 000 stop 0xc3e1e260 portmap 0xc3da8000 00000086 00000001 0 000 stop 0xc3da8260 rpc.ugidd 0xc3d8c000 00000093 00000001 0 000 stop 0xc3d8c260 scanlogd 0xc3d58000 00000099 00000001 0 000 run 0xc3d58260 syslogd 0xc3d2a000 00000103 00000001 0 000 run 0xc3d2a260 klogd 0xc3cea000 00000122 00000001 0 000 stop 0xc3cea260 keyserv 0xc3c8e000 00000141 00000001 0 000 run 0xc3c8e260 rpc.mountd 0xc3c54000 00000144 00000001 0 000 run 0xc3c54260 rpc.nfsd 0xc3c28000 00000154 00000001 0 000 stop 0xc3c28260 atd 0xc3b78000 00000163 00000001 0 000 stop 0xc3b78260 inetd 0xc3c4a000 00000165 00000001 0 000 stop 0xc3c4a260 gpm 0xc3ad0000 00000181 00000001 0 000 stop 0xc3ad0260 lpd 0xc3b5a000 00000194 00000001 0 000 stop 0xc3b5a260 sendmail 0xc3e5e000 00000201 00000001 0 000 stop 0xc3e5e260 sshd 0xc3a16000 00000205 00000001 0 000 stop 0xc3a16260 cron more> 0xc396a000 00000212 00000001 0 000 run 0xc396a260 nscd 0xc396e000 00000213 00000212 0 000 run 0xc396e260 nscd 0xc393e000 00000214 00000213 0 000 run 0xc393e260 nscd 0xc3938000 00000215 00000213 0 000 run 0xc3938260 nscd 0xc3a44000 00000217 00000213 0 000 stop 0xc3a44260 nscd 0xc3b40000 00000218 00000213 0 000 stop 0xc3b40260 nscd 0xc39f2000 00000219 00000213 0 000 stop 0xc39f2260 nscd 0xc1ef0000 00000235 00000001 0 000 stop 0xc1ef0260 login 0xc1ec2000 00000236 00000001 0 000 stop 0xc1ec2260 login 0xc7db4000 00000237 00000001 0 000 stop 0xc7db4260 mingetty 0xc7e90000 00000238 00000001 0 000 stop 0xc7e90260 mingetty 0xc3e86000 00000239 00000001 0 000 stop 0xc3e86260 mingetty 0xc7dc8000 00000240 00000001 0 000 stop 0xc7dc8260 mingetty 0xc3882000 00000241 00000235 0 000 stop 0xc3882260 bash 0xc3f1c000 00000250 00000236 0 000 stop 0xc3f1c260 bash 0xc4d54000 00000257 00000250 0 000 run 0xc4d54260*vmstat 0xc4c4c000 00000258 00000241 0 000 run 0xc4c4c260 swptst 0xc4ea4000 00000259 00000258 0 000 run 0xc4ea4260 swptst kdb> btp 257 EBP EIP Function(args) 0xc012c720 page_launder+0xa8 (0x7, 0x1) 0xc4d55f00 0xc012d15f do_try_to_free_pages+0x47 (0x7, 0x1) 0xc4d55f10 0xc012d3d6 try_to_free_pages+0x22 (0x7) 0xc4d55f30 0xc012dfe7 __alloc_pages+0x1db (0xc02fc5f4, 0x0) 0xc4d55f40 0xc012e099 __get_free_pages+0x29 (0x7, 0x0) 0xc4d55f68 0xc012ac40 kmem_cache_grow+0xc4 (0xc1e25d40, 0x7) 0xc4d55f84 0xc012adef kmem_cache_alloc+0x4f (0xc1e25d40, 0x7) 0xc4d55fa0 0xc013b5b9 getname+0x25 (0x8049972) 0xc4d55fbc 0xc0131104 sys_open+0x1c (0x8049972, 0x10800, 0xbffff62c, 0x8049972, 0x3cb) 0xc010aa74 system_call+0x3c kdb> go Entering kdb (0xc4d54000) due to Keyboard Entry kdb> bt EBP EIP Function(args) 0xc4d55ee4 0xc012c889 page_launder+0x211 (0x7, 0x1) 0xc4d55f00 0xc012d15f do_try_to_free_pages+0x47 (0x7, 0x1) 0xc4d55f10 0xc012d3d6 try_to_free_pages+0x22 (0x7) 0xc4d55f30 0xc012dfe7 __alloc_pages+0x1db (0xc02fc5f4, 0x0) 0xc4d55f40 0xc012e099 __get_free_pages+0x29 (0x7, 0x0) 0xc4d55f68 0xc012ac40 kmem_cache_grow+0xc4 (0xc1e25d40, 0x7) 0xc4d55f84 0xc012adef kmem_cache_alloc+0x4f (0xc1e25d40, 0x7) 0xc4d55fa0 0xc013b5b9 getname+0x25 (0x8049972) 0xc4d55fbc 0xc0131104 sys_open+0x1c (0x8049972, 0x10800, 0xbffff62c, 0x8049972, 0x3cb) 0xc010aa74 system_call+0x3c kdb> set LOGGING 0 SysRq: Show Memory Mem-info: Free pages: 1400kB ( 0kB HighMem) ( Active: 2383, inactive_dirty: 17, inactive_clean: 0, free: 350 (255 510 765) ) 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB = 512kB) 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 888kB) = 0kB) Swap cache: add 27827, delete 25738, find 2485/8960 Free swap: 166256kB 32752 pages of RAM 0 pages of HIGHMEM 4325 reserved pages 295 pages shared 2089 pages swap cached 0 pages in page table cache Buffer memory: 68kB SysRq: Show Memory Mem-info: Free pages: 1400kB ( 0kB HighMem) ( Active: 2383, inactive_dirty: 17, inactive_clean: 0, free: 350 (255 510 765) ) 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB = 512kB) 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 888kB) = 0kB) Swap cache: add 27827, delete 25738, find 2485/8960 Free swap: 166256kB 32752 pages of RAM 0 pages of HIGHMEM 4325 reserved pages 295 pages shared 2089 pages swap cached 0 pages in page table cache Buffer memory: 68kB SysRq: Show Memory Mem-info: Free pages: 1400kB ( 0kB HighMem) ( Active: 2367, inactive_dirty: 33, inactive_clean: 0, free: 350 (255 510 765) ) 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB = 512kB) 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 888kB) = 0kB) Swap cache: add 27827, delete 25738, find 2485/8960 Free swap: 166256kB 32752 pages of RAM 0 pages of HIGHMEM 4325 reserved pages 295 pages shared 2089 pages swap cached 0 pages in page table cache Buffer memory: 68kB SysRq: Show Memory Mem-info: Free pages: 1400kB ( 0kB HighMem) ( Active: 2375, inactive_dirty: 24, inactive_clean: 0, free: 350 (255 510 765) ) 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB = 512kB) 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 888kB) = 0kB) Swap cache: add 27827, delete 25738, find 2485/8960 Free swap: 166256kB 32752 pages of RAM 0 pages of HIGHMEM 4325 reserved pages 295 pages shared 2089 pages swap cached 0 pages in page table cache Buffer memory: 68kB SysRq: Show Memory Mem-info: Free pages: 1400kB ( 0kB HighMem) ( Active: 2380, inactive_dirty: 20, inactive_clean: 0, free: 350 (255 510 765) ) 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB = 512kB) 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 888kB) = 0kB) Swap cache: add 27827, delete 25738, find 2485/8960 Free swap: 166256kB 32752 pages of RAM 0 pages of HIGHMEM 4325 reserved pages 295 pages shared 2089 pages swap cached 0 pages in page table cache Buffer memory: 68kB SysRq: Show Memory Mem-info: Free pages: 1400kB ( 0kB HighMem) ( Active: 2367, inactive_dirty: 32, inactive_clean: 0, free: 350 (255 510 765) ) 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB = 512kB) 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 888kB) = 0kB) Swap cache: add 27827, delete 25738, find 2485/8960 Free swap: 166256kB 32752 pages of RAM 0 pages of HIGHMEM 4325 reserved pages 296 pages shared 2089 pages swap cached 0 pages in page table cache Buffer memory: 68kB SysRq: Terminate All Tasks SysRq: Terminate All Tasks Unable to handle kernel NULL pointer dereference at virtual address 00000000 c02fc18c *pde = 00000000 Oops: 0002 CPU: 0 EIP: 0010:[] EFLAGS: 00010246 eax: 00000000 ebx: c4d54000 ecx: c034f540 edx: 00000000 esi: c02fc600 edi: 00000001 ebp: c4d55f10 esp: c4d55ee4 ds: 0018 es: 0018 ss: 0018 Process vmstat (pid: 257, stackpage=c4d55000) Stack: 00000023 00000000 c02fc600 01234567 c4d54000 00000000 00000000 01234567 c4d54000 00000000 00000000 c4d55f30 c012df42 00000000 00000000 00000007 c1e25d40 00000007 00000001 c4d55f40 c012e099 c02fc5f4 00000000 c4d55f68 Call Trace: [] (0) [] (32) [] (16) [] (40) [] (28) [] (28) [] (28) [] Code: 00 00 00 00 c8 bf f9 c7 c8 bf f9 c7 00 00 00 00 9c c1 2f c0 >>EIP; c02fc18c <===== Trace; c012df42 <__alloc_pages+136/264> Trace; c012e099 <__get_free_pages+29/38> Trace; c012ac40 Trace; c012adef Trace; c013b5b9 Trace; c0131104 Trace; c010aa74 trimmed profile of frozen trace buffer: Total entries: 1048555 Total usecs: 306733.07 %TOTAL-TIME TOTAL-USECS AVG/CALL CALLS ADDR FUNC 0.0169% 51.97 7.42 7 c0117bf3 schedule 0.0260% 79.87 1.90 42 c012113c update_one_process 0.0361% 110.77 2.64 42 c012102d update_wall_time_one_tick 0.0389% 119.33 2.44 49 c010e6a8 mask_and_ack_8259A 0.1040% 319.13 7.60 42 c010f2a2 timer_interrupt 0.2113% 648.25 0.12 5400 c012cfa3 refill_inactive 0.2427% 744.51 0.14 5400 c012d3c1 try_to_free_pages 0.2476% 759.32 0.14 5400 c012cf4e inactive_shortage 0.2601% 797.69 0.15 5400 c014495f prune_icache 0.2648% 812.23 0.15 5400 c012d128 do_try_to_free_pages 0.2667% 818.12 0.15 5400 c0143444 prune_dcache 0.2903% 890.30 0.16 5400 c01447ef dispose_list 0.2970% 911.15 0.17 5400 c013500a wakeup_bdflush 0.3303% 1013.14 0.19 5400 c012d2fe wakeup_kswapd 0.3831% 1175.24 0.22 5400 c0144a49 shrink_icache_memory 0.4028% 1235.43 0.23 5400 c01437ed shrink_dcache_memory 0.4935% 1513.78 0.14 10800 c012aa3a kmem_cache_shrink 0.5372% 1647.67 0.31 5400 c012c68b page_launder 0.7191% 2205.76 0.41 5400 c0118c16 wake_up_process 0.8518% 2612.77 0.24 10800 c012a9e2 __kmem_cache_shrink 0.9255% 2838.67 0.13 21600 c012cea7 free_shortage 1.1119% 3410.50 0.21 16200 c012dd64 __alloc_pages_limit 1.4108% 4327.35 0.40 10802 c0117fff __wake_up 1.4517% 4452.75 0.16 27000 c012e1aa nr_inactive_clean_pages 1.5890% 4874.09 0.90 5400 c01444db sync_all_inodes 1.8562% 5693.61 0.53 10800 c012a982 is_chained_kmem_cache 2.8923% 8871.67 0.18 48600 c012e16e nr_free_pages 5.3308% 16351.35 0.19 86400 c012cdb3 refill_inactive_scan 5.7669% 17689.02 0.20 86436 c012b45d age_page_down_nolock 8.2360% 25262.57 2.34 10800 c012b08b kmem_cache_reap 11.9917% 36782.49 0.43 86436 c012b4eb deactivate_page_nolock 14.7300% 45181.80 0.25 183604 c0134ebb try_to_free_buffers 17.3333% 53167.09 0.30 178200 c0134e5b sync_page_buffers 19.2186% 58949.66 0.32 183604 c012e0f9 __free_pages