* LTP VM test slower under 2.6.3 than 2.4.20
@ 2004-02-20 16:36 Kirk True
2004-02-20 16:39 ` Martin J. Bligh
` (2 more replies)
0 siblings, 3 replies; 14+ messages in thread
From: Kirk True @ 2004-02-20 16:36 UTC (permalink / raw)
To: kernelnewbies, Linux-MM
Hi all,
Executing the LTP "mem01" VM test shows a huge time discrepancy between
2.4.20 and 2.6.3. Under 2.4.20 the total time is around 5 seconds, while
under 2.6.3 the system seems to hang for nearly a minute.
Where in particular should I start to look to see if it's a
configuration/environment issue or a real problem? What other
information would be helpful to know?
Thanks greatly!!!
Kirk
--------------
2.6.3:
# time /tmp/ltp-full-20040206/testcases/kernel/mem/mem/mem01
Free Mem: 749 Mb
Free Swap: 1992 Mb
Total Free: 2741 Mb
Total Tested: 1024 Mb
mem01 0 INFO : touching 1024MB of malloc'ed memory (linear)
mem01 1 PASS : malloc - alloc of 1024MB succeeded
real 0m53.134s
user 0m0.066s
sys 0m3.292s
2.4.20:
# time /tmp/ltp-full-20040206/testcases/kernel/mem/mem/mem01
Free Mem: 859 Mb
Free Swap: 1992 Mb
Total Free: 2852 Mb
Total Tested: 1024 Mb
mem01 0 INFO : touching 1024MB of malloc'ed memory (linear)
mem01 1 PASS : malloc - alloc of 1024MB succeeded
real 0m5.493s
user 0m0.090s
sys 0m1.260s
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-20 16:36 LTP VM test slower under 2.6.3 than 2.4.20 Kirk True
@ 2004-02-20 16:39 ` Martin J. Bligh
2004-02-20 18:57 ` Kirk True
[not found] ` <40363DEE.9040608@movaris.com>
2004-02-20 22:45 ` Nick Piggin
2004-02-23 7:19 ` Andrew Morton
2 siblings, 2 replies; 14+ messages in thread
From: Martin J. Bligh @ 2004-02-20 16:39 UTC (permalink / raw)
To: Kirk True, kernelnewbies, Linux-MM
> Hi all,
>
> Executing the LTP "mem01" VM test shows a huge time discrepancy between 2.4.20 and 2.6.3. Under 2.4.20 the total time is around 5 seconds, while under 2.6.3 the system seems to hang for nearly a minute.
>
> Where in particular should I start to look to see if it's a configuration/environment issue or a real problem? What other information would be helpful to know?
>
> Thanks greatly!!!
> Kirk
A kernel profile might help.
M.
> --------------
>
> 2.6.3:
>
># time /tmp/ltp-full-20040206/testcases/kernel/mem/mem/mem01
> Free Mem: 749 Mb
> Free Swap: 1992 Mb
> Total Free: 2741 Mb
> Total Tested: 1024 Mb
> mem01 0 INFO : touching 1024MB of malloc'ed memory (linear)
> mem01 1 PASS : malloc - alloc of 1024MB succeeded
>
> real 0m53.134s
> user 0m0.066s
> sys 0m3.292s
>
>
>
> 2.4.20:
>
># time /tmp/ltp-full-20040206/testcases/kernel/mem/mem/mem01
> Free Mem: 859 Mb
> Free Swap: 1992 Mb
> Total Free: 2852 Mb
> Total Tested: 1024 Mb
> mem01 0 INFO : touching 1024MB of malloc'ed memory (linear)
> mem01 1 PASS : malloc - alloc of 1024MB succeeded
>
> real 0m5.493s
> user 0m0.090s
> sys 0m1.260s
>
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-20 16:39 ` Martin J. Bligh
@ 2004-02-20 18:57 ` Kirk True
2004-02-20 19:03 ` Kirk True
[not found] ` <40363DEE.9040608@movaris.com>
1 sibling, 1 reply; 14+ messages in thread
From: Kirk True @ 2004-02-20 18:57 UTC (permalink / raw)
To: Martin J. Bligh; +Cc: kernelnewbies, Linux-MM
Hi all,
> A kernel profile might help.
Here's the profiles comparing the two versions. I don't understand why
no "do_page_fault" shows up under 2.4.20 or why "gunzip" does.
Kirk
-------------------------------------------------
2.6.3:
# readprofile -r;./mem01;readprofile -m /boot/System.map-2.6.3 \
>> captured_profile2.6.3
1446 poll_idle 24.9310
4 delay_tsc 0.1667
71 do_page_fault 0.0536
41 schedule 0.0238
13 __might_sleep 0.0637
1 prepare_to_wait 0.0068
1 put_files_struct 0.0042
3229 do_softirq 16.3081
1 run_timer_softirq 0.0022
4807 total 0.0262
2.4.20:
# readprofile -r;./mem01;readprofile -m /boot/System.map-2.4.20 \
>> captured_profile2.4.20
347 gunzip 0.1559
8 acpi_restore_state_mem 0.0021
2 proc_dostring 0.0033
3 access_process_vm 0.0054
2 __mod_timer 0.0038
1 del_timer 0.0065
1 update_one_process 0.0036
1 notifier_chain_unregister 0.0052
365 total 0.0020
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-20 18:57 ` Kirk True
@ 2004-02-20 19:03 ` Kirk True
0 siblings, 0 replies; 14+ messages in thread
From: Kirk True @ 2004-02-20 19:03 UTC (permalink / raw)
To: Kirk True; +Cc: Martin J. Bligh, kernelnewbies, Linux-MM
OK, try this profile for 2.4.20 instead (helps to use the right System.map):
221 default_idle 4.6042
2 error_code 0.0333
1 restore_fpu 0.0312
12 apm_bios_call_simple 0.0833
6 do_page_fault 0.0047
1 do_anonymous_page 0.0039
1 pte_alloc 0.0057
2 set_page_dirty 0.0179
1 add_to_page_cache_unique 0.0078
2 mark_page_accessed 0.0417
1 kmem_cache_free 0.0208
1 activate_page 0.0078
1 lru_cache_add 0.0104
2 __lru_cache_del 0.0179
14 shrink_cache 0.0179
3 swap_out_pmd 0.0117
2 try_to_swap_out 0.0048
2 __free_pages_ok 0.0030
7 rmqueue 0.0129
1 try_to_free_buffers 0.0039
1 ide_dmaproc 0.0012
53 fast_clear_page 0.6625
337 total 0.0003
Kirk
Kirk True wrote:
> Hi all,
>
> > A kernel profile might help.
>
> Here's the profiles comparing the two versions. I don't understand why
> no "do_page_fault" shows up under 2.4.20 or why "gunzip" does.
>
> Kirk
>
> -------------------------------------------------
>
>
>
> 2.6.3:
>
> # readprofile -r;./mem01;readprofile -m /boot/System.map-2.6.3 \
> >> captured_profile2.6.3
>
> 1446 poll_idle 24.9310
> 4 delay_tsc 0.1667
> 71 do_page_fault 0.0536
> 41 schedule 0.0238
> 13 __might_sleep 0.0637
> 1 prepare_to_wait 0.0068
> 1 put_files_struct 0.0042
> 3229 do_softirq 16.3081
> 1 run_timer_softirq 0.0022
> 4807 total 0.0262
>
>
>
> 2.4.20:
>
> # readprofile -r;./mem01;readprofile -m /boot/System.map-2.4.20 \
> >> captured_profile2.4.20
>
> 347 gunzip 0.1559
> 8 acpi_restore_state_mem 0.0021
> 2 proc_dostring 0.0033
> 3 access_process_vm 0.0054
> 2 __mod_timer 0.0038
> 1 del_timer 0.0065
> 1 update_one_process 0.0036
> 1 notifier_chain_unregister 0.0052
> 365 total 0.0020
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <40363DEE.9040608@movaris.com>]
* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-20 16:36 LTP VM test slower under 2.6.3 than 2.4.20 Kirk True
2004-02-20 16:39 ` Martin J. Bligh
@ 2004-02-20 22:45 ` Nick Piggin
2004-02-21 2:50 ` Kirk True
2004-02-21 15:39 ` Kirk True
2004-02-23 7:19 ` Andrew Morton
2 siblings, 2 replies; 14+ messages in thread
From: Nick Piggin @ 2004-02-20 22:45 UTC (permalink / raw)
To: Kirk True; +Cc: kernelnewbies, Linux-MM
Kirk True wrote:
> Hi all,
>
> Executing the LTP "mem01" VM test shows a huge time discrepancy
> between 2.4.20 and 2.6.3. Under 2.4.20 the total time is around 5
> seconds, while under 2.6.3 the system seems to hang for nearly a minute.
>
> Where in particular should I start to look to see if it's a
> configuration/environment issue or a real problem? What other
> information would be helpful to know?
>
2.6 must start writeout, does it?
Can you post vmstat 1 logs for each kernel?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-20 22:45 ` Nick Piggin
@ 2004-02-21 2:50 ` Kirk True
2004-02-21 2:50 ` Kirk True
2004-02-21 15:39 ` Kirk True
1 sibling, 1 reply; 14+ messages in thread
From: Kirk True @ 2004-02-21 2:50 UTC (permalink / raw)
To: Nick Piggin; +Cc: kernelnewbies, Linux-MM
Nick Piggin wrote:
> 2.6 must start writeout, does it?
Sorry, but how can I tell?
> Can you post vmstat 1 logs for each kernel?
The 2.4.20 vmstat is attached (formatting inline is ugly) but I couldn't
get a vmstat for 2.6.3. Running strace vmstat shows that it's dying when
reading from /proc/stat with a SEGFAULT. I googled about this but didn't
see anything.
Kirk
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-20 22:45 ` Nick Piggin
2004-02-21 2:50 ` Kirk True
@ 2004-02-21 15:39 ` Kirk True
1 sibling, 0 replies; 14+ messages in thread
From: Kirk True @ 2004-02-21 15:39 UTC (permalink / raw)
To: Nick Piggin; +Cc: kernelnewbies, Linux-MM
[-- Attachment #1: Type: text/plain, Size: 122 bytes --]
> Can you post vmstat 1 logs for each kernel?
Attached is the vmstat output. The CPU stats are pretty interesting.
Kirk
[-- Attachment #2: vmstatcombined.txt --]
[-- Type: text/plain, Size: 2929 bytes --]
2.4.20:
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy wa id
0 0 0 886312 8612 66152 0 0 0 0 109 88 1 0 0 99
0 0 0 886312 8612 66152 0 0 0 0 132 140 1 0 0 99
2 0 1016 5272 4212 28188 0 368 36 368 115 380 13 65 0 22
1 0 45116 5236 4192 27872 0 40912 4 40912 724 1889 0 22 0 78
1 1 76604 5236 4192 27872 0 30912 0 31004 646 1416 0 17 0 83
0 2 112316 5236 4200 27872 0 36876 12 36872 675 873 0 14 0 86
0 0 3560 930120 4192 27872 0 21536 8 21540 492 487 1 13 0 86
0 0 3304 930372 4192 27876 0 0 0 0 124 45 0 0 0 100
0 0 3304 930372 4192 27876 0 0 0 0 107 24 0 0 0 100
2.6.3:
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy wa id
0 0 41884 862460 752 11916 0 0 0 0 1008 67 0 0 0 100
1 0 41884 692660 756 11912 0 0 0 0 1028 115 9 38 0 53
1 0 41884 254580 756 11912 0 0 0 0 1003 15 3 97 0 0
0 12 52536 4148 128 2232 1124 74012 2128 74184 16321 930 2 60 38 0
4 11 112492 4104 152 2460 180 1856 332 1856 5281 23 0 0 100 0
4 11 131836 4936 152 2728 600 19344 1084 19348 5782 196 0 32 68 0
6 8 151528 4512 172 2624 388 19692 672 19704 5650 134 0 33 67 0
3 10 172484 4688 176 3056 436 20956 1188 20960 5718 200 0 30 70 0
4 10 191760 4448 176 3036 220 19276 668 19288 5533 135 0 53 47 0
5 11 212456 4580 180 3056 416 20696 960 20704 5656 155 0 33 67 0
4 10 231376 4256 180 3176 124 18920 488 18924 5429 145 7 50 43 0
0 7 41996 880800 208 4340 496 124 1740 136 1177 174 0 28 72 0
0 4 41996 878752 216 5608 900 0 2180 0 1092 164 0 2 98 0
0 4 41996 876576 224 6880 896 0 2192 0 1094 174 0 0 100 0
0 2 41996 874280 236 8032 1148 0 2308 0 1138 241 0 2 98 0
0 3 41996 871976 248 9220 1044 0 2236 4 1121 226 0 2 98 0
0 0 41996 870696 264 9728 768 0 1284 48 1074 161 7 1 39 52
0 0 41996 870696 264 9728 0 0 0 0 1024 42 0 0 0 100
0 0 41996 870700 264 9728 0 0 0 0 1003 17 0 0 0 100
0 0 41996 870700 264 9728 0 0 0 0 1023 40 0 0 0 100
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-20 16:36 LTP VM test slower under 2.6.3 than 2.4.20 Kirk True
2004-02-20 16:39 ` Martin J. Bligh
2004-02-20 22:45 ` Nick Piggin
@ 2004-02-23 7:19 ` Andrew Morton
2004-02-23 16:51 ` Kirk True
2 siblings, 1 reply; 14+ messages in thread
From: Andrew Morton @ 2004-02-23 7:19 UTC (permalink / raw)
To: Kirk True; +Cc: kernelnewbies, Linux-MM
Kirk True <ktrue@movaris.com> wrote:
>
> Executing the LTP "mem01" VM test shows a huge time discrepancy between
> 2.4.20 and 2.6.3. Under 2.4.20 the total time is around 5 seconds, while
> under 2.6.3 the system seems to hang for nearly a minute.
I'd be wondering if your disk system is correctly running in DMA mode.
On my 256MB test box, mem01 takes 31 seconds under 2.4.25, 25 seconds
under 2.6.3-mm3.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-23 7:19 ` Andrew Morton
@ 2004-02-23 16:51 ` Kirk True
2004-02-23 19:42 ` Andrew Morton
2004-02-24 15:08 ` Kirk True
0 siblings, 2 replies; 14+ messages in thread
From: Kirk True @ 2004-02-23 16:51 UTC (permalink / raw)
To: Andrew Morton; +Cc: kernelnewbies, Linux-MM
Hi Andrew,
> I'd be wondering if your disk system is correctly running in DMA mode.
Apparently support for my hardware isn't magically preset in 2.6.3 as it
was somehow in 2.4. After including it in the kernel the values returned
by hdparm -Tt /dev/hda/ were sped up by a factor of 10! Thanks!
But...
> On my 256MB test box, mem01 takes 31 seconds under 2.4.25, 25 seconds
> under 2.6.3-mm3.
...I'm still seeing a factor of two speed slowdown between 2.4.20 and
2.6.3. Would it help to do a vmstat log/graph for the new results?
Kirk
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-23 16:51 ` Kirk True
@ 2004-02-23 19:42 ` Andrew Morton
2004-02-24 15:08 ` Kirk True
1 sibling, 0 replies; 14+ messages in thread
From: Andrew Morton @ 2004-02-23 19:42 UTC (permalink / raw)
To: Kirk True; +Cc: kernelnewbies, Linux-MM
Kirk True <ktrue@movaris.com> wrote:
>
> Hi Andrew,
>
> > I'd be wondering if your disk system is correctly running in DMA mode.
>
> Apparently support for my hardware isn't magically preset in 2.6.3 as it
> was somehow in 2.4. After including it in the kernel the values returned
> by hdparm -Tt /dev/hda/ were sped up by a factor of 10! Thanks!
>
> But...
>
> > On my 256MB test box, mem01 takes 31 seconds under 2.4.25, 25 seconds
> > under 2.6.3-mm3.
>
> ...I'm still seeing a factor of two speed slowdown between 2.4.20 and
> 2.6.3. Would it help to do a vmstat log/graph for the new results?
>
Try 2.6.3-mm3. It's tons better at this.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-23 16:51 ` Kirk True
2004-02-23 19:42 ` Andrew Morton
@ 2004-02-24 15:08 ` Kirk True
2004-02-24 23:38 ` Andrew Morton
1 sibling, 1 reply; 14+ messages in thread
From: Kirk True @ 2004-02-24 15:08 UTC (permalink / raw)
To: kernelnewbies, Linux-MM
Hi all,
I just upgraded to 2.6.3-mm2 but am still seeing a factor of two speed
slowdown between 2.4.20 and 2.6.3-mm2 for both sequential and random
memory accesses into 1024 MB allocated from malloc.
I'm not trying to whine, I'm looking to explain this behavior and maybe
be of some help somehow :)
Kirk
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: LTP VM test slower under 2.6.3 than 2.4.20
2004-02-24 15:08 ` Kirk True
@ 2004-02-24 23:38 ` Andrew Morton
0 siblings, 0 replies; 14+ messages in thread
From: Andrew Morton @ 2004-02-24 23:38 UTC (permalink / raw)
To: Kirk True; +Cc: kernelnewbies, Linux-MM
Kirk True <kirk@movaris.com> wrote:
>
> I just upgraded to 2.6.3-mm2 but am still seeing a factor of two speed
> slowdown between 2.4.20 and 2.6.3-mm2 for both sequential and random
> memory accesses into 1024 MB allocated from malloc.
2.6 VM has problems, but is usually OK for single-task stuff.
You'd need to tell us a lot about your machine, and provide sufficient
information for others to reproduce what you're seeing.
And run some other tests to verify that your disk system is achieving the
same bandwidth under both kernels. Not `hdparm -t' please, it is crap.
Something like
time (dd if=/dev/zero of=/mnt/x/foo bs=1M count=2000 ; sync)
umount /mnt/x
mount /mnt/x
time dd if=/mnt/x/foo of=/dev/null bs=1M
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2004-02-24 23:38 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2004-02-20 16:36 LTP VM test slower under 2.6.3 than 2.4.20 Kirk True
2004-02-20 16:39 ` Martin J. Bligh
2004-02-20 18:57 ` Kirk True
2004-02-20 19:03 ` Kirk True
[not found] ` <40363DEE.9040608@movaris.com>
[not found] ` <5760000.1077296989@[10.10.2.4]>
2004-02-22 15:32 ` Kirk True
2004-02-20 22:45 ` Nick Piggin
2004-02-21 2:50 ` Kirk True
2004-02-21 2:50 ` Kirk True
2004-02-21 15:39 ` Kirk True
2004-02-23 7:19 ` Andrew Morton
2004-02-23 16:51 ` Kirk True
2004-02-23 19:42 ` Andrew Morton
2004-02-24 15:08 ` Kirk True
2004-02-24 23:38 ` Andrew Morton
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox