* large mem, heavy paging issues (256M VmStk on Athlon) @ 2001-02-21 0:39 Eric Whiting 2001-02-21 22:39 ` Rik van Riel 0 siblings, 1 reply; 5+ messages in thread From: Eric Whiting @ 2001-02-21 0:39 UTC (permalink / raw) To: linux-mm I'm working with an application in Lisp. It runs on a Solaris box and uses about 1.3G of RAM and 9M stack before it exits after 2hours of running. I have been trying to run the same application on linux. It's memory usage hits about 1.2G and then it loses it's brain. Under 2.4.2pre4 PIII SMP 800Mhz 1.2G physical 2G swap -- it seg faults with a stack size of 6M. Under 2.4.2pre4 Athlon 1000Mhz 1.2G physical 2G swap -- it seg faults with a stack size of 256M. None of the boxes are OC. Both have big memory set to 64G. This problem is either 1. an application problem 2. a linux vm/mm problem 3. a wacky HW problem. 4. ??? I keep dumps of /proc/$pid/status and /proc/$pid/maps to watch memory usage. (10s intervals for the 2 hours it takes before the seg fault happens). I even captured a full strace of the run one time. I have not been able to figure out what might be wrong. I think this might just be an application problem, but wanted to know if this sort of problem has been seen before. The box does go crazy paging once I run it out of physical ram. Questions: ---------- Anyone have ideas? What other things can I do? Why the athlon/PIII difference? I'm dumping 2G of physical RAM into the PIII box tomorrow and I'll see what happens in the absense of the killer paging. Thanks, eric Last valid status (for Athlon T-bird test): ------------------------------------------- Name: access State: R (running) Pid: 597 PPid: 596 TracerPid: 0 Uid: 500 500 500 500 Gid: 100 100 100 100 FDSize: 256 Groups: 100 VmSize: 1839028 kB VmLck: 0 kB VmRSS: 1057196 kB VmData: 1553132 kB VmStk: 252376 kB <---- looks like we are in big trouble!!!! VmExe: 12 kB VmLib: 4196 kB SigPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 8000000002000000 SigCgt: 00000000004154db CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 Last valid status (for PIII test): ---------------------------------- Name: access (no it is not the M$ access) State: R (running) Pid: 1205 PPid: 1203 TracerPid: 0 Uid: 15041 15041 15041 15041 Gid: 21 21 21 21 FDSize: 256 Groups: 21 42 181 41 VmSize: 1830076 kB VmLck: 0 kB VmRSS: 1159196 kB VmData: 1790132 kB VmStk: 6744 kB VmExe: 12 kB VmLib: 3896 kB SigPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 8000000002000000 SigCgt: 00000000004154db CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 Last valid maps output (for PIII) ------------------------- 08048000-0804b000 r-xp 00000000 00:0c 29261935 /home/pendsm1/access/bin11/linux/access 0804b000-0804d000 rw-p 00002000 00:0c 29261935 /home/pendsm1/access/bin11/linux/access 0804d000-0805a000 rwxp 00000000 00:00 0 40000000-40013000 r-xp 00000000 03:03 275293 /lib/ld-2.1.3.so 40013000-40014000 rw-p 00012000 03:03 275293 /lib/ld-2.1.3.so 40014000-40015000 r-xp 00000000 00:0c 29261965 /home/pendsm1/access/bin11/linux/climxm.so 40015000-40016000 rw-p 00000000 00:0c 29261965 /home/pendsm1/access/bin11/linux/climxm.so 4001d000-4001e000 rw-p 00000000 00:00 0 4001e000-4003a000 r-xp 00000000 03:03 275303 /lib/libm.so.6 4003a000-4003b000 rw-p 0001b000 03:03 275303 /lib/libm.so.6 4003b000-4003d000 r-xp 00000000 03:03 275302 /lib/libdl.so.2 4003d000-4003f000 rw-p 00001000 03:03 275302 /lib/libdl.so.2 4003f000-4011a000 r-xp 00000000 03:03 275298 /lib/libc.so.6 4011a000-4011f000 rw-p 000da000 03:03 275298 /lib/libc.so.6 4011f000-40122000 rw-p 00000000 00:00 0 40122000-40186000 r-xp 00000000 00:0c 29261971 /home/pendsm1/access/bin11/linux/libacl601.so 40186000-40192000 rw-p 00063000 00:0c 29261971 /home/pendsm1/access/bin11/linux/libacl601.so 40192000-401ac000 rw-p 00000000 00:00 0 401ac000-401b4000 r-xp 00000000 03:03 275309 /lib/libnss_files.so.2 401b4000-401b5000 rw-p 00007000 03:03 275309 /lib/libnss_files.so.2 401b5000-402c0000 r-xp 00000000 03:03 1151085 /usr/X11R6/LessTif/Motif1.2/lib/libXm.so.1.0.2 402c0000-402dc000 rw-p 0010a000 03:03 1151085 /usr/X11R6/LessTif/Motif1.2/lib/libXm.so.1.0.2 402dc000-402de000 rw-p 00000000 00:00 0 402de000-402eb000 r-xp 00000000 03:03 1167687 /usr/X11R6/lib/libXpm.so.4.11 402eb000-402ec000 rw-p 0000c000 03:03 1167687 /usr/X11R6/lib/libXpm.so.4.11 402ec000-402f9000 r-xp 00000000 03:03 1167677 /usr/X11R6/lib/libXext.so.6.4 402f9000-402fa000 rw-p 0000c000 03:03 1167677 /usr/X11R6/lib/libXext.so.6.4 402fa000-40343000 r-xp 00000000 03:03 1167689 /usr/X11R6/lib/libXt.so.6.0 40343000-40347000 rw-p 00048000 03:03 1167689 /usr/X11R6/lib/libXt.so.6.0 40347000-40348000 rw-p 00000000 00:00 0 40348000-40412000 r-xp 00000000 03:03 1167669 /usr/X11R6/lib/libX11.so.6.1 40412000-40417000 rw-p 000c9000 03:03 1167669 /usr/X11R6/lib/libX11.so.6.1 40417000-40418000 rw-p 00000000 00:00 0 40418000-40420000 r-xp 00000000 03:03 1167667 /usr/X11R6/lib/libSM.so.6.0 40420000-40422000 rw-p 00007000 03:03 1167667 /usr/X11R6/lib/libSM.so.6.0 40422000-40437000 r-xp 00000000 03:03 1167663 /usr/X11R6/lib/libICE.so.6.3 40437000-40438000 rw-p 00014000 03:03 1167663 /usr/X11R6/lib/libICE.so.6.3 40438000-4043a000 rw-p 00000000 00:00 0 50000000-51444000 rwxp 0000a000 00:0c 29261937 /home/pendsm1/access/bin11/linux/access-11-0r.dxl 51444000-5144a000 rwxp 00000000 00:00 0 5144a000-51848000 rwxp 0144e000 00:0c 29261937 /home/pendsm1/access/bin11/linux/access-11-0r.dxl 51848000-5184a000 rwxp 00000000 00:00 0 5184a000-51ac0000 rwxp 0184c000 00:0c 29261937 /home/pendsm1/access/bin11/linux/access-11-0r.dxl 51ac0000-51aca000 rwxp 00000000 00:00 0 51aca000-51c58000 rwxp 01ac2000 00:0c 29261937 /home/pendsm1/access/bin11/linux/access-11-0r.dxl 51c58000-51eca000 rwxp 00000000 00:00 0 51eca000-51ed4000 rwxp 01c50000 00:0c 29261937 /home/pendsm1/access/bin11/linux/access-11-0r.dxl 51ed4000-56898000 rwxp 00000000 00:00 0 56898000-57246000 rwxp 00000000 00:00 0 57246000-57bf4000 rwxp 00000000 00:00 0 57bf4000-582f4000 rwxp 00000000 00:00 0 582f4000-58834000 rwxp 00000000 00:00 0 58834000-58db4000 rwxp 00000000 00:00 0 58db4000-597f4000 rwxp 00000000 00:00 0 597f4000-5a5f4000 rwxp 00000000 00:00 0 5a5f4000-5b334000 rwxp 00000000 00:00 0 5b334000-5bdb4000 rwxp 00000000 00:00 0 5bdb4000-5c034000 rwxp 00000000 00:00 0 5c034000-5cd74000 rwxp 00000000 00:00 0 5cd74000-5e4f4000 rwxp 00000000 00:00 0 5e4f4000-5fb34000 rwxp 00000000 00:00 0 5fb34000-62674000 rwxp 00000000 00:00 0 62674000-65674000 rwxp 00000000 00:00 0 65674000-66e74000 rwxp 00000000 00:00 0 66e74000-67234000 rwxp 00000000 00:00 0 67234000-69234000 rwxp 00000000 00:00 0 69234000-6c574000 rwxp 00000000 00:00 0 6c574000-6ef74000 rwxp 00000000 00:00 0 6ef74000-70134000 rwxp 00000000 00:00 0 70134000-737b4000 rwxp 00000000 00:00 0 737b4000-77fb4000 rwxp 00000000 00:00 0 77fb4000-7b274000 rwxp 00000000 00:00 0 7b274000-7ce34000 rwxp 00000000 00:00 0 7ce34000-7f2b4000 rwxp 00000000 00:00 0 7f2b4000-82634000 rwxp 00000000 00:00 0 82634000-846f4000 rwxp 00000000 00:00 0 846f4000-bef5a000 rwxp 2cb00000 00:00 0 bf000000-bf008000 rwxp 01c68000 00:0c 29261937 /home/pendsm1/access/bin11/linux/access-11-0r.dxl bf008000-bf0fa000 rwxp 00000000 00:00 0 bf96a000-c0000000 rwxp ff96b000 00:00 0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux.eu.org/Linux-MM/ ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: large mem, heavy paging issues (256M VmStk on Athlon) 2001-02-21 0:39 large mem, heavy paging issues (256M VmStk on Athlon) Eric Whiting @ 2001-02-21 22:39 ` Rik van Riel 2001-02-23 20:12 ` Eric Whiting 0 siblings, 1 reply; 5+ messages in thread From: Rik van Riel @ 2001-02-21 22:39 UTC (permalink / raw) To: Eric Whiting; +Cc: linux-mm On Tue, 20 Feb 2001, Eric Whiting wrote: > I'm working with an application in Lisp. It runs on a Solaris > box and uses about 1.3G of RAM and 9M stack before it exits > after 2hours of running. > > I have been trying to run the same application on linux. It's > memory usage hits about 1.2G and then it loses it's brain. > This problem is either > 1. an application problem > 2. a linux vm/mm problem > 3. a wacky HW problem. > 4. ??? It's a glibc problem in combination with an oddity in the Linux VM layer. At 1GB, Linux starts with the mmap() areas, so brk() will only work up to 1GB. When going over that, glibc's malloc() should use mmap() instead to get more memory... > What other things can I do? > Last valid maps output (for PIII) > ------------------------- > 08048000-0804b000 r-xp 00000000 00:0c 29261935 /home/pendsm1/access/bin11/linux/access > 0804b000-0804d000 rw-p 00002000 00:0c 29261935 /home/pendsm1/access/bin11/linux/access > 0804d000-0805a000 rwxp 00000000 00:00 0 > 40000000-40013000 r-xp 00000000 03:03 275293 /lib/ld-2.1.3.so ^^^^^^^^ Mapped at 1GB, so brk() will hit this point... regards, Rik -- Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com.br/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux.eu.org/Linux-MM/ ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: large mem, heavy paging issues (256M VmStk on Athlon) 2001-02-21 22:39 ` Rik van Riel @ 2001-02-23 20:12 ` Eric Whiting 2001-02-24 2:38 ` Rik van Riel 0 siblings, 1 reply; 5+ messages in thread From: Eric Whiting @ 2001-02-23 20:12 UTC (permalink / raw) To: Rik van Riel; +Cc: linux-mm Rick, Thanks for the info -- but I'm not sure I understand what the fix is to be. Does my lisp engine need to be recompiled with a newer glibc? Do I need to change something else? I think the strace showed the process is using mainly malloc (mmap) for memory allocation. I do see some brk() calls at the first. (these appear to be returning a 2G number not a 1G number like you suggested) access_test/ruby> egrep -i 'brk' strace.log brk(0) = 0x8051a48 brk(0) = 0x8051a48 brk(0x8051e60) = 0x8051e60 brk(0x8052000) = 0x8052000 brk(0x8054000) = 0x8054000 brk(0x8055000) = 0x8055000 brk(0x8056000) = 0x8056000 brk(0x8059000) = 0x8059000 brk(0x805a000) = 0x805a000 And later: (mainly mmap calls) (I assume the mmap calls of interest are the ones that contain MAP_ANONYMOUS which come from a malloc call) old_mmap(0x4011f000, 11804, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4011f000 old_mmap(0x40192000, 104095, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40192000 old_mmap(0xbf008000, 991232, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xbf008000 old_mmap(0x51444000, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x51444000 old_mmap(0x51ed4000, 77348864, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x51ed4000 old_mmap(0x568a0000, 10092544, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x568a0000 old_mmap(0x57246000, 10149888, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x57246000 HERE is a successful malloc of 1.7G old_mmap(0x57bf4000, 1731616768, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x57bf4000 old_mmap(0x402dc000, 7260, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x402dc000 old_mmap(0x40347000, 648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40347000 old_mmap(0x40417000, 124, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40417000 old_mmap(0x40438000, 6864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40438000 old_mmap(0x56898000, 10149888, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x56898000 old_mmap(0x57246000, 10149888, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x57246000 old_mmap(0x57bf4000, 7340032, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x57bf4000 old_mmap(0x582f4000, 5505024, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x582f4000 old_mmap(0x58834000, 5767168, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x58834000 old_mmap(0x58db4000, 10747904, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x58db4000 <SNIP> A ton of these follow as the program runs. Is there something I can change on the linux box? Or is this something the application has to change? Thanks, eric Rik van Riel wrote: > > On Tue, 20 Feb 2001, Eric Whiting wrote: > > > I'm working with an application in Lisp. It runs on a Solaris > > box and uses about 1.3G of RAM and 9M stack before it exits > > after 2hours of running. > > > > I have been trying to run the same application on linux. It's > > memory usage hits about 1.2G and then it loses it's brain. > > > This problem is either > > 1. an application problem > > 2. a linux vm/mm problem > > 3. a wacky HW problem. > > 4. ??? > > It's a glibc problem in combination with an oddity in the Linux > VM layer. > > At 1GB, Linux starts with the mmap() areas, so brk() will only > work up to 1GB. When going over that, glibc's malloc() should > use mmap() instead to get more memory... > > > What other things can I do? > > > Last valid maps output (for PIII) > > ------------------------- > > 08048000-0804b000 r-xp 00000000 00:0c 29261935 /home/pendsm1/access/bin11/linux/access > > 0804b000-0804d000 rw-p 00002000 00:0c 29261935 /home/pendsm1/access/bin11/linux/access > > 0804d000-0805a000 rwxp 00000000 00:00 0 > > 40000000-40013000 r-xp 00000000 03:03 275293 /lib/ld-2.1.3.so > ^^^^^^^^ > > Mapped at 1GB, so brk() will hit this point... > > regards, > > Rik > -- > Virtual memory is like a game you can't win; > However, without VM there's truly nothing to lose... > > http://www.surriel.com/ > http://www.conectiva.com/ http://distro.conectiva.com.br/ -- __________________________________________________________________ Eric T. Whiting AMI Semiconductors (208) 234-6717 2300 Buckskin Road (208) 234-6659 (fax) Pocatello,ID 83201 ewhiting@poci.amis.com -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux.eu.org/Linux-MM/ ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: large mem, heavy paging issues (256M VmStk on Athlon) 2001-02-23 20:12 ` Eric Whiting @ 2001-02-24 2:38 ` Rik van Riel 2001-02-24 3:15 ` Eric Whiting 0 siblings, 1 reply; 5+ messages in thread From: Rik van Riel @ 2001-02-24 2:38 UTC (permalink / raw) To: Eric Whiting; +Cc: linux-mm On Fri, 23 Feb 2001, Eric Whiting wrote: > Thanks for the info -- but I'm not sure I understand what the fix is > to be. Does my lisp engine need to be recompiled with a newer glibc? > Do I need to change something else? If your lisp engine is dynamically linked to glibc, a simple glibc upgrade should do the trick (if this thing is fixed in newer glibcs). > I think the strace showed the process is using mainly malloc (mmap) > for memory allocation. I do see some brk() calls at the first. (these > appear to be returning a 2G number not a 1G number like you suggested) > brk(0x805a000) = 0x805a000 Actually, this would be 0x0805a000 if you wrote out the leading 0 ... this is more like 128 MB ;) regards, Rik -- Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com.br/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux.eu.org/Linux-MM/ ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: large mem, heavy paging issues (256M VmStk on Athlon) 2001-02-24 2:38 ` Rik van Riel @ 2001-02-24 3:15 ` Eric Whiting 0 siblings, 0 replies; 5+ messages in thread From: Eric Whiting @ 2001-02-24 3:15 UTC (permalink / raw) To: Rik van Riel; +Cc: linux-mm Rik van Riel wrote: > > If your lisp engine is dynamically linked to glibc, a simple > glibc upgrade should do the trick (if this thing is fixed in > newer glibcs). > > > I think the strace showed the process is using mainly malloc (mmap) > > for memory allocation. I do see some brk() calls at the first. (these > > appear to be returning a 2G number not a 1G number like you suggested) > > > brk(0x805a000) = 0x805a000 > > Actually, this would be 0x0805a000 if you wrote out the leading > 0 ... this is more like 128 MB ;) oops -- yes I need to count digits better. The mmaps look ok however: HERE is a successful malloc of 1.7G old_mmap(0x57bf4000, 1731616768, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x57bf4000 I'll go back to the application and work on this from some other angles. Thanks for the sanity check and suggestions. eric > > regards, > > Rik > -- > Virtual memory is like a game you can't win; > However, without VM there's truly nothing to lose... > > http://www.surriel.com/ > http://www.conectiva.com/ http://distro.conectiva.com.br/ > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux.eu.org/Linux-MM/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux.eu.org/Linux-MM/ ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2001-02-24 3:15 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2001-02-21 0:39 large mem, heavy paging issues (256M VmStk on Athlon) Eric Whiting 2001-02-21 22:39 ` Rik van Riel 2001-02-23 20:12 ` Eric Whiting 2001-02-24 2:38 ` Rik van Riel 2001-02-24 3:15 ` Eric Whiting
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox