Re: Looking for better VM

Re: Looking for better VM

[Jesse Pollard]

------
> On Wed, 8 Nov 2000, Szabolcs Szakacsits wrote:
> > On Mon, 6 Nov 2000, Rik van Riel wrote:
[snip]
> > You could ask, so what's the point for non-overcommit if we use
> > process killing in the end? And the answer, in *practise* this almost
> > never happens, root can always clean up and no processes are lost
> > [just as when disk is "full" except the reserved area for root]. See?
> > Human get a chance against hard-wired AI.
> > 
> > I also didn't say non-overcommit should be used as default and a
> > patch http://www.cs.helsinki.fi/linux/linux-kernel/2000-13/1208.html,
> > developed for 2.3.99-pre3 by Eduardo Horvath and unfortunately was
> > ignored completely, implemented it this way. 
> 
> OK. This is a lot more reasonable. I'm actually looking
> into putting non-overcommit as a configurable option in
> the kernel.
> 
> However, this does not save you from the fact that the
> system is essentially deadlocked when nothing can get
> more memory and nothing goes away. Non-overcommit won't
> give you any extra reliability unless your applications
> are very well behaved ... in which case you don't need
> non-overcommit.

Applications are not usually the problem, users are. If a user starts
one "well behaved" process, and then starts another, and another....
The system WILL go OOM, and with unpredictable results (as far as the user
is concerned).

The Eduardo Horvath patch works exactly as he designed. It allowed overcommit
by root, disallowed user generating overcommit. or it could disallow
overcommit by all, or operate the same as without the patch (but it did
accumulate some statistics).

The problem is that unless user memory resource controls are available to
the administrator to establish some policy, system deadlock will always
occur, OR you have random shutdowns, or random process aborts. The resource
controls should allow an administrator defined policy, established in user
space, and enforced by the kernel. The kernel should be able to enforce any
policy from no memory restriction (current, and reasonable for single user
workstations), to fully disabled overcommit (dedicated multi-user batch
processing in clustered environments).

I know the patch was an early prototype. It did provide some identification
of the locations that resource controls could/should be done (this should be a
2.5 developement item).

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Looking for better VM

[Szabolcs Szakacsits]

On Mon, 6 Nov 2000, Rik van Riel wrote:
> On Mon, 6 Nov 2000, Szabolcs Szakacsits wrote:
> > On Wed, 1 Nov 2000, Rik van Riel wrote:
> > > but simply because 
> > > it appears there has been amazingly little research on this 
> > > subject and it's completely unknown which approach will work 
> > There has been lot of research, this is the reason most Unices support
> > both non-overcommit and overcommit memory handling default to
> > non-overcommit [think of reliability and high availability].
> It's a shame you didn't take the trouble to actually
> go out and see that non-overcommit doesn't solve the
> "out of memory" deadlock problem.

Read my *entire* email again and please try to understand. No deadlock
at all since kernel *falls back* to process killing if memory reserved
for *root* is also out.

You could ask, so what's the point for non-overcommit if we use
process killing in the end? And the answer, in *practise* this almost
never happens, root can always clean up and no processes are lost
[just as when disk is "full" except the reserved area for root]. See?
Human get a chance against hard-wired AI.

I also didn't say non-overcommit should be used as default and a
patch http://www.cs.helsinki.fi/linux/linux-kernel/2000-13/1208.html,
developed for 2.3.99-pre3 by Eduardo Horvath and unfortunately was
ignored completely, implemented it this way. 

And with a runtime tunable OOM killer, Linux really would beat the
competitors [where it is quite behind at present] in this area. See?
Human get a chance against hard-wired AI again.

Believe me, there are people [don't read only kernel lists] who wants
a reliable and controllable system and where the kernel doesn't play
Russan rulet.

[who missed my first email: forget about mem quotas and the the
non-scalable "add GB's of swap" in this discussion].

> [if you want an explanation, look in the archives,
> we've explained this a dozen times now]
 
I've been reading the list much longer than you and really pissed of
that after so many years of discussions, this problem and user
requirements^Wwishes are still not understood. You think black and
white but the world is colorful.

	Szaka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Looking for better VM

[Rik van Riel]

On Wed, 8 Nov 2000, Szabolcs Szakacsits wrote:
> On Mon, 6 Nov 2000, Rik van Riel wrote:
> > On Mon, 6 Nov 2000, Szabolcs Szakacsits wrote:
> > > On Wed, 1 Nov 2000, Rik van Riel wrote:
> > > > but simply because 
> > > > it appears there has been amazingly little research on this 
> > > > subject and it's completely unknown which approach will work 
> > > There has been lot of research, this is the reason most Unices support
> > > both non-overcommit and overcommit memory handling default to
> > > non-overcommit [think of reliability and high availability].
> > It's a shame you didn't take the trouble to actually
> > go out and see that non-overcommit doesn't solve the
> > "out of memory" deadlock problem.
> 
> Read my *entire* email again and please try to understand. No deadlock
> at all since kernel *falls back* to process killing if memory reserved
> for *root* is also out.
> 
> You could ask, so what's the point for non-overcommit if we use
> process killing in the end? And the answer, in *practise* this almost
> never happens, root can always clean up and no processes are lost
> [just as when disk is "full" except the reserved area for root]. See?
> Human get a chance against hard-wired AI.
> 
> I also didn't say non-overcommit should be used as default and a
> patch http://www.cs.helsinki.fi/linux/linux-kernel/2000-13/1208.html,
> developed for 2.3.99-pre3 by Eduardo Horvath and unfortunately was
> ignored completely, implemented it this way. 

OK. This is a lot more reasonable. I'm actually looking
into putting non-overcommit as a configurable option in
the kernel.

However, this does not save you from the fact that the
system is essentially deadlocked when nothing can get
more memory and nothing goes away. Non-overcommit won't
give you any extra reliability unless your applications
are very well behaved ... in which case you don't need
non-overcommit.

regards,

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.

http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Looking for better VM

[Mikulas Patocka]

Hi.

> > I also didn't say non-overcommit should be used as default and a
> > patch http://www.cs.helsinki.fi/linux/linux-kernel/2000-13/1208.html,
> > developed for 2.3.99-pre3 by Eduardo Horvath and unfortunately was
> > ignored completely, implemented it this way. 
> 
> OK. This is a lot more reasonable. I'm actually looking
> into putting non-overcommit as a configurable option in
> the kernel.
> 
> However, this does not save you from the fact that the
> system is essentially deadlocked when nothing can get
> more memory and nothing goes away. Non-overcommit won't
> give you any extra reliability unless your applications
> are very well behaved ... in which case you don't need
> non-overcommit.

BTW. Why does your OOM killer in 2.4 try to kill process that mmaped most
memory? mmap is hamrless. mmap on files can't eat memory and swap.

Imagine a case: you have database server that mmaps the whole 2G file but
doesn't have too much anonymous memory. You have an offending process that
does while (1) malloc(1000) and fills up 512M swap. Your OOM killer would
kill the server first...

Mikulas

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Looking for better VM

[Christoph Rohland]

Hi Mikulas,

On Wed, 8 Nov 2000, Mikulas Patocka wrote:
> BTW. Why does your OOM killer in 2.4 try to kill process that mmaped
> most memory? mmap is hamrless. mmap on files can't eat memory and
> swap.

Be careful: They may have shm segments mmaped!

Greetings
		Christoph

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Looking for better VM

[Ingo Oeser]

On Wed, Nov 08, 2000 at 05:36:40PM +0100, Mikulas Patocka wrote:
> BTW. Why does your OOM killer in 2.4 try to kill process that mmaped most
> memory? mmap is hamrless. mmap on files can't eat memory and swap.

Don't complain, build your own and test it ;-)

Apply my patch

http://www.tu-chemnitz.de/~ioe/oom_kill_api.patch

and install your own OOM handler using install_oom_killer() 
from <linux/swap.h>. It has all the needed documentation inline
that will be build along the kernel-api-book.

Have fun researching in this area.

PS: Applies cleanly since oom_kill.c exists and also against
   2.4.0-test11-pre1.

Regards

Ingo Oeser
-- 
To the systems programmer, users and applications
serve only to provide a test load.
<esc>:x
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Looking for better VM

[Rik van Riel]

On Wed, 8 Nov 2000, Mikulas Patocka wrote:

> BTW. Why does your OOM killer in 2.4 try to kill process that mmaped
> most memory? mmap is hamrless. mmap on files can't eat memory and
> swap.

Because the thing is too stupid to take that into
consideration? :)

Btw, if your mmap()ed file still takes 1GB of memory,
you have 1GB of freeable memory left and you shouldn't
be out of memory ... or should you??

regards,

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.

http://www.conectiva.com/		http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/