From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <3911B131.4A565CE0@sgi.com> Date: Thu, 04 May 2000 10:19:45 -0700 From: Rajagopal Ananthanarayanan MIME-Version: 1.0 Subject: Re: Oops in __free_pages_ok (pre7-1) (Long) (backtrace) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: Linus Torvalds Cc: Kanoj Sarcar , linux-mm@kvack.org, "David S. Miller" , Rik van Riel List-ID: Linus Torvalds wrote: > > On Thu, 4 May 2000, Rajagopal Ananthanarayanan wrote: > > > > I did some testing of this patch with dbench. > > The kernel starts shooting processes down pretty quickly > > ("VM: killing process XXX") on a 2 CPU 64MB system, > > with nothing but dbench (8 clients). A concurrently > > running vmstat shows very low free memory with some swapping, > > and the buffer space remaining around 50MB. [ ... ] > > Rik, I think the kswapd logic is wrong, and I suspect you made it > worsewhen you added the while-loop. The problem looks like that while > kswapd is working on one zone, it will entirely ignore any other zones. I > think the logic should be more like [ ... ] > What do you think? I suspect that the added do-loop in pre7 just made the > "lost wakeups" problem worse by concentrating on one zone for a longer > while and thus more likely to lose wakeups for lower zones (because it > already looked at those). > > There might be other details like this lurking, but this looks like a good > first try. Ananth, willing to give it a whirl? > > Linus I haven't looked at the code, but I replaced the whole while (1) loop with the new for(;;). Things still remain the same: when running dbench VM starts killing processes. Following is a vmstat trace during the dbench run with the latest change to kswapd. For some reason, the system seems to swap more with the change; I've attached a second vmstat trace as of early AM today before the change. Again, I have a 2 CPU box with 64M memory ... the tests are very controlled, nothing changes except what I want to change (kernel bits). ------------- vmstat trace with kswapd change ---------------------- [root@delilah /root]# vmstat 1 1000 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 0 41664 968 11144 0 0 12 2 55 14 1 1 98 0 0 0 0 41664 968 11144 0 0 0 0 106 13 0 0 100 0 0 0 0 41664 968 11144 0 0 0 0 105 8 0 0 100 0 0 0 0 41664 968 11144 0 0 0 0 104 8 0 0 100 0 0 0 0 41664 968 11144 0 0 0 0 107 26 0 0 100 0 0 0 0 41664 968 11144 0 0 0 0 110 36 0 0 100 0 0 0 0 41664 968 11144 0 0 0 0 114 46 0 0 100 0 0 0 0 41236 968 11144 0 0 0 0 108 42 0 0 100 0 0 0 0 41216 968 11144 0 0 0 0 105 23 1 0 99 0 8 0 0 38324 1036 13260 0 0 142 85 203 356 2 5 93 0 8 1 0 24376 1096 25548 0 0 33 3378 239 705 1 21 78 0 8 1 0 21340 1116 28220 0 0 26 2482 304 1960 0 7 92 1 7 1 0 11540 1160 36896 0 0 41 2623 318 5384 1 17 82 0 8 1 0 6212 1180 41568 0 0 20 2813 332 3467 1 8 91 0 8 1 0 3040 1192 44360 0 0 6 2727 271 803 0 7 92 0 8 2 432 1208 1148 46484 0 432 60 6968 754 4099 0 7 92 0 8 2 648 1112 1164 46780 0 216 14 3249 348 2720 0 9 91 0 8 0 3948 1240 1112 49984 0 3300 44 22247 2210 6100 0 5 95 0 8 0 4336 1540 1076 49896 0 388 28 4597 317 222 1 12 87 3 5 1 6132 1200 444 51400 1736 3892 1872 31316 5098 2307 0 15 85 0 7 0 6612 1092 452 51528 172 908 329 7872 862 463 0 21 79 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 5 1 6944 1548 456 51148 192 1052 339 8263 1362 405 1 14 86 2 4 1 7212 1532 436 51216 16 668 35 7667 678 470 0 24 76 2 6 1 7072 1540 436 51136 360 808 224 8202 1170 756 0 21 79 1 3 2 7048 1260 448 51320 28 328 61 4082 484 126 0 20 80 2 1 3 7012 964 460 51720 24 180 46 2545 310 90 0 7 93 1 4 1 6984 992 472 51664 24 192 29 2917 320 91 1 17 82 0 1 5 7064 1508 468 51116 32 248 28 2062 345 85 1 16 83 0 3 3 6956 1540 456 51004 32 288 188 4072 525 183 0 14 85 0 5 1 7168 1516 464 50864 84 628 454 4657 824 249 1 10 89 0 7 0 7500 1108 496 51204 20 672 204 2668 412 171 4 9 87 2 5 0 7484 756 504 51476 0 404 179 3601 558 312 1 6 93 0 7 0 7476 1244 488 51076 4 228 342 1557 402 194 1 5 94 0 6 1 7456 1244 496 51076 4 120 691 1530 399 118 3 5 92 0 7 0 7372 1416 488 50780 28 344 1068 1086 351 178 1 6 93 2 4 0 7352 1448 500 50740 0 244 893 506 413 152 2 3 95 0 6 0 7252 1200 500 50944 48 124 1411 289 225 166 0 7 93 0 7 0 7324 1404 488 50796 72 124 708 531 455 526 1 1 98 0 7 0 7364 1480 488 50804 12 224 858 225 965 892 1 4 96 0 6 1 7284 1424 500 50736 16 240 1986 140 218 192 0 10 89 2 4 0 7420 876 500 51476 4 352 1424 588 599 511 2 10 88 1 6 0 7424 1356 496 50944 68 168 679 1042 695 337 1 3 96 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 6 0 7312 1228 520 50936 44 176 513 2044 476 210 4 8 88 0 6 0 7408 1464 516 50812 12 264 135 1566 277 94 3 12 85 1 5 0 7384 1228 536 50952 0 120 208 2530 312 147 1 6 92 0 6 0 7512 1332 528 50996 0 212 104 1553 326 104 0 10 89 0 6 0 7548 1544 536 50908 16 236 132 2059 430 110 1 7 92 2 4 1 7740 1360 552 51156 64 540 427 4635 614 232 3 6 91 0 6 0 7564 1204 552 51172 56 124 217 1031 333 98 3 3 94 0 6 0 7460 1064 560 51112 12 116 718 1529 423 195 6 7 87 0 6 0 7516 1212 580 50968 16 236 662 559 258 176 2 7 91 0 6 0 7472 1448 596 50580 16 0 638 1000 232 182 5 10 85 1 5 1 7556 1380 608 50700 16 124 354 531 306 147 2 5 93 1 5 0 7604 1204 624 50912 12 180 712 2545 470 139 3 6 91 1 5 0 7612 1372 636 50688 8 292 1082 2573 507 199 2 6 92 0 6 0 7600 1516 652 50548 4 168 392 1042 264 125 5 4 90 0 6 0 7592 1356 676 50756 0 108 442 527 286 157 5 8 87 0 6 0 7488 3400 704 48556 0 0 659 0 213 186 4 2 93 0 6 0 7392 7188 712 44660 0 0 745 0 221 220 11 13 76 0 6 0 7392 2588 724 49252 0 0 302 3000 306 486 14 13 73 0 6 2 7420 2004 732 49944 0 136 130 1292 292 3150 7 7 85 5 2 0 7520 2040 732 49988 0 104 38 2268 333 357 1 2 97 4 3 1 7808 1144 724 51068 8 408 371 7555 844 3960 4 8 88 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 4 2 2 8116 1480 728 50852 24 692 190 5143 664 335 3 6 92 1 5 0 7860 1516 736 50608 24 100 122 3083 452 228 3 9 88 0 6 0 7760 1132 744 50780 4 100 352 44 423 99 3 3 94 4 2 0 7676 2516 744 49292 0 128 610 1032 271 189 15 15 70 3 1 5 8352 852 756 51380 28 1112 624 11778 1640 3164 3 9 88 4 2 0 7928 1420 760 50480 8 264 225 1566 483 306 7 10 83 0 6 0 7876 4504 764 47252 0 0 425 1500 328 144 10 16 74 0 6 0 7876 10068 764 41664 0 0 265 2000 418 124 19 21 60 2 4 1 7876 6608 764 45124 0 0 208 3137 266 1170 18 26 55 0 6 1 7872 1712 764 50000 0 0 255 1943 327 4264 8 14 78 3 3 2 7864 1540 764 50364 0 204 21 2003 357 3479 7 16 77 0 6 2 7716 888 764 50992 0 68 147 2071 306 3040 8 10 83 2 4 2 7804 900 772 51016 0 272 270 3982 599 3871 8 14 78 1 5 1 7688 7028 780 44668 0 8 120 2416 326 1194 25 25 50 0 6 1 7688 3152 780 48540 0 0 189 2364 335 10084 17 39 45 0 6 1 7688 1352 780 50332 0 0 185 1843 381 6073 16 20 64 4 2 1 7708 2636 776 49132 0 120 40 4975 298 2135 6 12 82 4 2 2 7832 3856 792 48100 0 348 259 8521 714 6546 6 10 84 4 2 1 7816 5792 792 46028 4 152 145 5785 377 1006 13 16 71 1 5 1 7804 2348 792 49380 52 0 223 859 247 4483 13 19 69 1 5 1 7828 2624 792 49156 0 112 104 2323 314 3228 11 15 74 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 2 4 1 7760 6600 792 44908 0 0 204 2878 361 1783 17 21 61 2 5 1 7736 4428 792 47224 220 0 297 3065 417 4822 27 38 35 0 7 1 7728 5128 792 46468 60 0 102 1865 434 2128 7 8 85 4 3 1 7724 2768 792 48844 12 0 169 3404 498 7431 11 16 73 2 4 0 7696 4128 792 47472 0 116 132 7416 444 4061 13 24 63 1 5 1 7692 5016 792 46572 0 0 8 1500 220 63 11 12 76 0 6 0 7672 5000 792 46564 0 0 12 2000 224 102 8 8 84 5 1 0 7672 4720 792 46860 0 0 69 2000 258 796 7 12 80 0 6 1 7672 5080 792 46484 0 0 115 3114 302 1880 28 34 38 0 6 1 7672 4308 792 47256 0 0 7 1718 286 2861 4 10 86 0 6 1 7672 6528 792 45020 12 0 114 1461 330 1508 3 5 92 0 6 0 7672 9080 792 42468 0 0 7 707 296 233 0 3 97 0 6 0 7672 11948 792 39592 8 0 30 3500 317 182 11 14 75 0 6 0 7672 35396 792 16144 0 0 69 0 273 169 0 7 93 0 6 0 7672 39716 792 11824 0 0 119 0 223 246 0 4 96 0 1 0 7628 41828 792 10520 768 0 532 0 216 248 1 1 98 0 0 0 7616 41320 792 10956 320 0 212 0 124 44 0 1 99 0 0 0 7616 41320 792 10956 0 0 0 16 119 12 0 0 100 0 0 0 7616 40580 792 11252 196 0 159 0 127 69 1 0 99 0 0 0 7616 40568 792 11252 0 0 0 0 106 23 1 0 99 0 8 1 7616 28856 792 22556 0 0 50 1045 137 239 2 17 80 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 7 1 1 7616 19400 792 32076 0 0 5 4999 307 1382 1 17 83 0 8 1 7616 8604 792 42832 0 0 10 5133 312 1871 0 19 81 0 8 1 7616 4616 792 46816 0 0 9 4186 295 758 0 8 92 0 8 1 7616 1032 792 50404 0 0 5 3804 325 803 0 10 90 ----------------------------------------------------------------------------------- ------------------ vmstat trace as of pure 7-4 patch ---------------------------- [root@delilah /root]# vmstat 1 1000 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 0 45176 712 8144 0 0 62 8 71 60 3 3 94 0 0 0 0 45176 712 8144 0 0 0 0 106 12 0 0 100 0 1 0 0 45160 716 8144 0 0 1 0 106 14 0 0 100 0 0 0 0 45024 732 8252 0 0 34 59 140 49 0 1 99 0 0 0 0 45024 732 8252 0 0 0 0 121 8 0 0 100 0 0 0 0 45024 732 8252 0 0 0 0 104 10 0 0 100 0 0 0 0 45024 732 8252 0 0 0 0 121 76 0 0 100 0 0 0 0 45020 736 8252 0 0 6 0 117 55 1 0 98 0 2 0 0 44640 772 8324 0 0 27 65 157 84 1 0 98 0 1 0 0 43436 908 8900 0 0 182 0 231 240 5 1 94 1 0 0 0 40604 940 10584 0 0 402 0 173 150 17 2 81 1 0 0 0 40452 952 10896 0 0 84 0 117 80 42 4 54 1 0 0 0 39072 952 10980 0 0 4 0 108 41 50 2 49 1 0 0 0 40476 988 11264 0 0 111 13 146 115 26 3 70 1 0 0 0 40916 988 11196 0 0 8 0 110 76 47 2 50 0 0 0 0 41576 988 11220 0 0 0 0 108 51 17 2 80 0 0 0 0 41576 988 11220 0 0 0 0 104 8 0 0 100 0 0 0 0 41576 988 11220 0 0 0 0 103 8 0 0 100 0 0 0 0 41576 988 11220 0 0 0 1 108 18 0 0 100 0 0 0 0 41576 988 11220 0 0 0 0 115 48 0 0 100 0 0 0 0 41576 988 11220 0 0 0 0 110 32 0 0 100 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 0 41140 988 11220 0 0 0 0 110 43 0 1 99 0 0 0 0 41128 988 11220 0 0 0 0 111 25 0 0 100 8 0 0 0 30488 1072 20260 0 0 73 0 126 186 1 13 86 0 8 0 0 20308 1120 29148 0 0 4 8500 203 66 1 16 83 0 8 1 0 11632 1160 36792 0 0 30 4599 267 395 0 15 84 0 8 1 0 2240 1208 45052 0 0 8 5787 346 981 1 17 82 0 8 2 0 1228 736 46520 0 0 14 2806 298 735 1 17 82 0 8 3 0 1144 388 46648 0 0 13 5154 392 871 1 15 84 5 4 1 1068 1080 384 47832 0 1068 17 15921 1059 1134 0 11 89 8 0 2 2220 1456 404 48616 0 1152 9 5788 487 123 0 10 89 0 8 2 3044 1028 408 49572 0 840 5 4210 264 95 0 17 83 0 8 1 3540 1100 412 49964 0 500 2 8125 222 89 0 8 92 0 8 2 3852 1088 416 50188 0 312 8 3359 224 89 0 9 91 0 9 1 4048 1072 448 50208 84 204 65 770 350 168 1 22 77 0 8 2 4016 1132 460 50068 200 0 88 5464 376 808 1 31 67 0 8 3 4016 1072 476 50112 56 0 25 4052 363 668 0 11 89 0 8 3 4016 1100 484 50080 0 0 13 4529 356 583 0 11 89 0 8 2 4384 1032 452 50504 0 372 4 9637 345 569 1 10 89 0 8 0 4432 1536 448 50052 0 48 0 1923 219 206 0 2 98 0 8 2 5004 1540 444 50628 0 572 71 6836 416 86 0 7 92 1 3 6 5280 1188 432 51168 0 332 21 9390 714 6872 1 11 88 Killed [root@delilah /root]# vmstat 1 1000 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 4 0 5672 30508 760 21128 12 31 85 710 114 159 4 8 87 0 0 0 5608 31360 760 20740 436 0 172 121 210 227 0 2 98 0 0 0 5608 31360 760 20740 0 0 0 0 168 10 0 0 100 0 0 0 5608 31360 760 20740 0 0 0 0 104 8 0 0 100 0 0 0 5608 31360 760 20740 0 0 0 0 105 8 0 0 100 0 0 0 5592 31324 760 20756 28 0 11 0 112 24 0 0 100 0 0 0 5592 31308 760 20768 12 0 3 22 124 26 0 0 100 0 0 0 5592 31300 760 20776 8 0 2 0 108 20 0 0 100 0 0 0 5592 31300 760 20776 0 0 0 0 103 8 0 0 100 0 0 0 5592 30860 760 20780 4 0 1 0 106 32 0 0 99 0 0 0 5592 30848 760 20780 0 0 0 0 106 25 0 0 100 1 7 1 5592 16064 760 35156 0 0 63 2365 176 1593 1 23 76 0 8 1 5592 2608 760 48632 0 0 38 5350 448 3538 0 28 72 0 8 2 5820 1060 744 50576 0 252 14 4888 409 848 0 10 89 0 5 4 6196 1208 744 50760 0 384 2 10556 543 1439 0 4 96 2 4 3 6080 1120 520 51372 100 1308 399 30927 2351 6011 0 36 63 0 8 2 5792 1056 552 51092 416 0 389 4967 776 6246 0 41 59 5 3 3 5344 1540 564 50908 188 1052 597 22696 2076 2402 0 25 75 0 4 0 5308 1536 576 50724 100 208 731 5552 732 243 1 7 91 0 5 1 5300 1348 588 50900 120 180 968 7045 809 1056 2 12 87 0 5 0 5200 1356 584 50760 56 0 299 1500 416 108 1 2 96 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 5 2 5100 1524 544 50644 24 56 169 4347 301 406 2 11 87 0 5 0 5128 1392 556 50896 48 68 225 2184 398 198 3 4 93 0 3 1 5032 1368 560 50884 32 0 306 1000 334 109 1 5 94 1 2 0 5100 1048 572 51256 0 96 327 2024 266 104 1 8 91 0 3 0 5080 1376 556 50900 12 40 386 2510 277 90 2 11 86 0 3 0 5160 1332 580 50892 0 120 1105 2030 220 148 4 10 86 0 3 0 5108 1228 588 50944 0 0 286 2500 349 76 2 5 93 2 1 0 5108 2488 604 49672 0 0 645 0 251 162 10 6 84 0 3 0 5140 1028 604 51248 0 232 365 2058 262 148 10 20 71 0 3 0 5452 1172 604 51304 0 392 262 4098 342 160 1 4 95 0 3 0 5324 1160 600 51152 0 0 194 1500 455 67 5 7 87 1 2 0 5308 1500 608 50736 0 0 204 2000 310 83 5 7 88 2 1 1 5228 1260 612 51024 0 92 1408 2023 220 209 33 42 25 0 3 0 5200 1084 616 51148 0 0 5 2000 308 721 7 7 86 0 3 0 5004 3116 616 49120 0 4 0 2001 263 48 5 3 92 3 0 0 4956 4336 616 47868 0 0 588 1500 217 106 31 34 34 0 3 0 4848 1496 616 50676 0 0 145 2000 242 62 14 16 70 3 0 0 4848 3684 616 48500 0 0 385 500 219 64 23 26 51 0 3 0 4848 1280 612 50888 0 0 140 3500 212 64 16 14 70 3 0 0 4848 5464 612 46760 0 0 225 0 201 52 8 23 70 0 3 0 4848 1396 616 50768 0 0 293 3000 207 74 20 26 54 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 1 2 0 4848 4236 616 47936 0 0 0 1500 258 49 6 4 90 0 3 0 4848 1956 616 50216 0 0 310 2000 209 81 24 27 49 2 1 0 4820 1488 608 50692 0 0 196 1500 303 58 10 8 81 0 3 0 4820 3896 608 48308 8 0 281 1000 237 127 18 22 60 0 3 0 4816 33088 608 19068 0 0 178 0 224 261 1 9 90 0 0 0 4752 34680 608 17808 144 0 232 0 168 121 0 1 98 0 1 0 4748 34600 608 17888 4 0 79 40 138 24 0 1 99 0 0 0 4736 34584 608 17884 8 0 2 0 109 16 0 0 100 0 0 0 4736 33996 608 18036 32 0 134 0 131 66 1 1 98 0 1 0 4736 31912 608 20056 0 0 55 0 132 74 1 4 95 0 8 0 4736 15540 608 36272 0 0 72 10000 217 121 0 23 77 4 4 2 4736 5060 608 46716 0 0 9 1670 222 107 0 20 80 0 8 1 4736 2728 608 48856 0 0 22 1653 225 593 0 8 92 2 6 2 4876 1184 600 50744 0 244 35 10361 400 640 1 10 89 3 5 2 5084 1204 608 50908 0 272 93 9176 647 6848 1 7 92 Killed -------------------------------------------------------------------------------- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux.eu.org/Linux-MM/