Want to allocate almost all the memory with no swap

Want to allocate almost all the memory with no swap

[Simon Derr]

Hi,

I'm currently trying to run a high-performance bench on a cluster of PCs
under Linux. This bench is the Linpack test, and needs a lot of memory to
store a matrix of numbers. Linpack needs to allocate as much as 240 Megs
on a machine that has 256 Megs of RAM, but I have to be sure that the
memory used by linpack will never be swapped on the disk.

The bests results I have obtained so far are the following:

* Under Linux 2.4.2: 
 -With swap on, part of the memory used by my process
  is swapped out, which is what I do not want
 
 -With swap off:The memory allocation is OK, my process has all the memory
it needs and can use it. BUT : here is what I get when running top:


CPU states: 49.6% user, 50.3% system,  0.0% nice,  0.0% idle
Mem:   254692K av,  252596K used,    2096K free,       0K shrd    68Kbuff
Swap:       0K av,       0K used,       0K free                  880Kcached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
    3 root      16   0     0    0     0 DW      0 49.6  0.0  11:10 kswapd
 1142 admin     18   0  240M 240M     4 R       0 49.6 96.5   9:14 loop
    5 root       9   0     0    0     0 SW      0  0.5  0.0   0:12 bdflush
 1149 root      11   0   408  408   240 R       0  0.3  0.1   0:03 top



(loop is the name of my process)
kswapd uses half of my CPU, and even bdflush seems to eat some cpu.
I find a bit weird to see kswapd eat my CPU when I have no swap at all...

* Under Linux 2.2.17:
 -With swap on, same pb as under 2.4.2
 -With swap off, the memory allocation fails.

I tried to tune the values under /proc/sys/vm, with no success.

Any ideas ?

(Please CC me in the reply)

	Thanks,

		Simon.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[James A. Sutherland]

On Thu, 19 Apr 2001 17:39:23 +0200 (MEST), you wrote:

>Hi,
>
>I'm currently trying to run a high-performance bench on a cluster of PCs
>under Linux. This bench is the Linpack test, and needs a lot of memory to
>store a matrix of numbers. Linpack needs to allocate as much as 240 Megs
>on a machine that has 256 Megs of RAM, but I have to be sure that the
>memory used by linpack will never be swapped on the disk.

Call mlockall() to lock all your memory into physical RAM - there's a
flag to set which ensures all your future allocations are locked as
well. You should be left with 16 Mb of physical RAM free, plus swap,
so you should be able to do this as long as the machine isn't too
heavily loaded at the time - no running Netscape during benchmarks :-)


James.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[Simon Derr]

On Thu, 19 Apr 2001, James A. Sutherland wrote:

> On Thu, 19 Apr 2001 17:39:23 +0200 (MEST), you wrote:
> 
> >Hi,
> >
> >I'm currently trying to run a high-performance bench on a cluster of PCs
> >under Linux. This bench is the Linpack test, and needs a lot of memory to
> >store a matrix of numbers. Linpack needs to allocate as much as 240 Megs
> >on a machine that has 256 Megs of RAM, but I have to be sure that the
> >memory used by linpack will never be swapped on the disk.
> 
> Call mlockall() to lock all your memory into physical RAM - there's a
> flag to set which ensures all your future allocations are locked as
> well. You should be left with 16 Mb of physical RAM free, plus swap,
> so you should be able to do this as long as the machine isn't too
> heavily loaded at the time - no running Netscape during benchmarks :-)

Well, I have removed as many processes deamons as I could, and there are
not many left.
But under both 2.4.2 and 2.2.17 (with swap on)I get, when I run my
program:

mlockall: Cannot allocate memory

	Simon.
     

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[Kev]

> Well, I have removed as many processes deamons as I could, and there are
> not many left.
> But under both 2.4.2 and 2.2.17 (with swap on)I get, when I run my
> program:
> 
> mlockall: Cannot allocate memory

mlockall() requires root priviledges.
-- 
Kevin L. Mitchell <klmitch@mit.edu>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[James A. Sutherland]

On Thu, 19 Apr 2001 11:57:45 -0400, you wrote:

>> Well, I have removed as many processes deamons as I could, and there are
>> not many left.
>> But under both 2.4.2 and 2.2.17 (with swap on)I get, when I run my
>> program:
>> 
>> mlockall: Cannot allocate memory
>
>mlockall() requires root priviledges.

Uh... if calling mlockall() as a non-root user gives that error,
someone should fix the error returned...

Sorry, I should have mentioned that it required you to be root!


James.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[Simon Derr]

On Thu, 19 Apr 2001, Kev wrote:

> 
> > Well, I have removed as many processes deamons as I could, and there are
> > not many left.
> > But under both 2.4.2 and 2.2.17 (with swap on)I get, when I run my
> > program:
> > 
> > mlockall: Cannot allocate memory
> 
> mlockall() requires root priviledges.
Even when running the program as root I get this error.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[James A. Sutherland]

On Thu, 19 Apr 2001 17:58:38 +0200 (MEST), you wrote:

>On Thu, 19 Apr 2001, James A. Sutherland wrote:
>
>> On Thu, 19 Apr 2001 17:39:23 +0200 (MEST), you wrote:
>> 
>> >Hi,
>> >
>> >I'm currently trying to run a high-performance bench on a cluster of PCs
>> >under Linux. This bench is the Linpack test, and needs a lot of memory to
>> >store a matrix of numbers. Linpack needs to allocate as much as 240 Megs
>> >on a machine that has 256 Megs of RAM, but I have to be sure that the
>> >memory used by linpack will never be swapped on the disk.
>> 
>> Call mlockall() to lock all your memory into physical RAM - there's a
>> flag to set which ensures all your future allocations are locked as
>> well. You should be left with 16 Mb of physical RAM free, plus swap,
>> so you should be able to do this as long as the machine isn't too
>> heavily loaded at the time - no running Netscape during benchmarks :-)
>
>Well, I have removed as many processes deamons as I could, and there are
>not many left.
>But under both 2.4.2 and 2.2.17 (with swap on)I get, when I run my
>program:
>
>mlockall: Cannot allocate memory

Hrm? Can you trim the consumption a bit - try cutting a big chunk out,
like 64 Mb, and see if it works then?


James.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[Simon Derr]

> >Well, I have removed as many processes deamons as I could, and there are
> >not many left.
> >But under both 2.4.2 and 2.2.17 (with swap on)I get, when I run my
> >program:
> >
> >mlockall: Cannot allocate memory
> 
> Hrm? Can you trim the consumption a bit - try cutting a big chunk out,
> like 64 Mb, and see if it works then?
> 
If I ask much less memory it works.. but has no interest.

In fact I a call mlockall() _before_ doing my big malloc, it works even
when I ask 240 megs, but:
-Under 2.2.17, quickly the kernel kills my process
-Under 2.4.2, kswapd again eats the CPU:

Mem:   254692K av,  252868K used,    1824K free,       0K shrd,  88K buff
Swap:  313256K av,    5476K used,  307780K free             4204K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
    3 root      14   0     0    0     0 RW      0 47.4  0.0  20:50 kswapd
 1277 root      14   0  241M 241M   968 R       0 46.8 96.8   0:12 loop
    5 root       9   0     0    0     0 SW      0  5.3  0.0   0:23 bdflush
 1278 root      10   0   468  404   404 R       0  0.3  0.1   0:00 top


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[Simon Derr]

On Thu, 19 Apr 2001, Simon Derr wrote:

> If I ask much less memory it works.. but has no interest.
> 
> In fact I a call mlockall() _before_ doing my big malloc, it works even
> when I ask 240 megs, but:
> -Under 2.2.17, quickly the kernel kills my process
> -Under 2.4.2, kswapd again eats the CPU:
> 
> Mem:   254692K av,  252868K used,    1824K free,       0K shrd,  88K buff
> Swap:  313256K av,    5476K used,  307780K free             4204K cached
> 
>   PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
>     3 root      14   0     0    0     0 RW      0 47.4  0.0  20:50 kswapd
>  1277 root      14   0  241M 241M   968 R       0 46.8 96.8   0:12 loop
>     5 root       9   0     0    0     0 SW      0  5.3  0.0   0:23 bdflush
>  1278 root      10   0   468  404   404 R       0  0.3  0.1   0:00 top
> 

Actually this is what happens under 2.4.2 :
when I launch the program, during about one minute kswapd eats 50% cpu,
and bdflush takes 2-5% cpu,
One minute later approx, they both stop eating the cpu and my process gets
almost 100% of the cpu (a PIII 733).

The same happens if I kill and launch my program a second time.

Sorry for the pollution I bring to your mailing list...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[Rik van Riel]

On Thu, 19 Apr 2001, Simon Derr wrote:

> Actually this is what happens under 2.4.2 :
> when I launch the program, during about one minute kswapd eats 50% cpu,
> and bdflush takes 2-5% cpu,
> One minute later approx, they both stop eating the cpu and my process gets
> almost 100% of the cpu (a PIII 733).
>
> The same happens if I kill and launch my program a second time.
>
> Sorry for the pollution I bring to your mailing list...

No. Thanks for telling us.  It is good to know that kswapd
exhibits this strange behaviour. It's admiteddly not a high
priority thing to fix, but it IS something to keep in mind.

thanks,

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[Szabolcs Szakacsits]

On Thu, 19 Apr 2001, Rik van Riel wrote:
> On Thu, 19 Apr 2001, Simon Derr wrote:
> > Actually this is what happens under 2.4.2 :
> > when I launch the program, during about one minute kswapd eats 50% cpu,
> > and bdflush takes 2-5% cpu,
> > One minute later approx, they both stop eating the cpu and my process gets
> > almost 100% of the cpu (a PIII 733).
> >
> > The same happens if I kill and launch my program a second time.
> >
> [...] Thanks for telling us.  It is good to know that kswapd
> exhibits this strange behaviour. It's admiteddly not a high
> priority thing to fix, but it IS something to keep in mind.

Apparently it's already fixed in ac kernels. Could the reduced number of
wakeup_kswapd() calls the reason?

	Szaka

---------- Forwarded message ----------
Date: Thu, 26 Apr 2001 13:49:56 +0200 (MEST)
From: Simon Derr <Simon.Derr@imag.fr>
To: Szabolcs Szakacsits <szaka@f-secure.com>
Subject: Re: Want to allocate almost all the memory with no swap

On Fri, 20 Apr 2001, Szabolcs Szakacsits wrote:

> Could you try Alan Cox's latest prekernel? It has VM related changes and
> your feedbeck could be valuable.
> ftp://ftp.fi.kernel.org/pub/linux/kernel/v2.4/
>
> And maybe 2.4.3? But Alan's would be more interesting because there are
> more significant modifications.
> ftp://ftp.kernel.org/pub/linux/kernel/people/alan/2.4/

Hi,

I'm sorry I had no time this week to do theses tests.
I can report this :

when allocating 240 megs of memory, and using
mlockall(MCL_CURRENT|MCL_FUTURE)
on a 256Megs RAM machine :

Under Linux 2.4.2: kswapd eats half of the CPU. A few times it did so
for a minute approximatively, and then my process got all the CPU. But
some other times kswapd has been taking the CPU much much longer (I did
not see the end, I stopped the test before)

Under Linux 2.4.3-ac9 : It works great !
kswapd does not eat the CPU at all (well, almost : I see it taking 0.1%
CPU during one second, but 'top' is not a very accurate measure tool)

        Simon.





--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Want to allocate almost all the memory with no swap

[James A. Sutherland]

On Thu, 19 Apr 2001 18:46:00 +0200 (MEST), you wrote:

>> >Well, I have removed as many processes deamons as I could, and there are
>> >not many left.
>> >But under both 2.4.2 and 2.2.17 (with swap on)I get, when I run my
>> >program:
>> >
>> >mlockall: Cannot allocate memory
>> 
>> Hrm? Can you trim the consumption a bit - try cutting a big chunk out,
>> like 64 Mb, and see if it works then?
>> 
>If I ask much less memory it works.. but has no interest.
>
>In fact I a call mlockall() _before_ doing my big malloc, it works even
>when I ask 240 megs, but:
>-Under 2.2.17, quickly the kernel kills my process

Gagh?! What signal? Any oops/core/panic?

>-Under 2.4.2, kswapd again eats the CPU:

Does it eat it continually, or do you get it back after a while? You
SHOULD see it chewing up all the CPU until it has evicted 240 Mb worth
of pages, then going back to sleep...


James.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/