Re: oom_killer - Does not perform when stress-tested (system hangs)

Re: oom_killer - Does not perform when stress-tested (system hangs)

[Srikrishnan Sundararajan]

Here are a few issues and questions regarding oom_kill.

When there are lots of user processes each mallocing 1 MB and sleeping
forever without freeing, there is a possibility of oom_kill to kill a
critical system task or other processes run as root as long as such a
process qualifies with the highest "badness" value. While the algorithm
does reduce the score for any root process, it does not preclude the
selection of such a process for killing.
I tried to prevent non-root processes from occupying large amounts of
virtual memory by setting ulimit for virtual memory. When I go beyond this,
the user program fails with an a cannot allocate memory error. But this
limit does not take the actual current status into account. ie. Limit is
not say 95% of total memory etc.
I understand that we can allocate quota for hard disk space, there by
preventing non-root processes from occupying any more disk space beyond the
quota limit. For example,  we can set quota such that when the
hard-disk-space is 95% full, only root can occupy further space. Is there a
similar way to enforce the same for memory usage. This might ensure that
errant non-root processes cannot keep on allocating memory, thereby can
prevent the swap from getting full.
Another thought is can we exclude root processes from  the "badness"
calculation. This might ensure that at no time a root process is killed by
oom_kill. Or we can modify this such that as long as a non-root process is
there, no root processes will be killed by oom_kill.

Also the current oom_kill does not seem to always identify the offending
process and kill that.  Is there any way of either identifying a specific
offending process or identify such a user and kill all his processes?


Srikrishnan


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: oom_killer - Does not perform when stress-tested (system hangs)

[William Lee Irwin III]

On Tue, Sep 10, 2002 at 01:05:06PM +0530, Srikrishnan Sundararajan wrote:
> When there are lots of user processes each mallocing 1 MB and sleeping
> forever without freeing, there is a possibility of oom_kill to kill a
> critical system task or other processes run as root as long as such a
> process qualifies with the highest "badness" value. While the algorithm
> does reduce the score for any root process, it does not preclude the
> selection of such a process for killing.

The only exempt process is init (pid 1). root processes may offend the
system just as easily, and their death is survivable.


On Tue, Sep 10, 2002 at 01:05:06PM +0530, Srikrishnan Sundararajan wrote:
> I tried to prevent non-root processes from occupying large amounts of
> virtual memory by setting ulimit for virtual memory. When I go beyond this,
> the user program fails with an a cannot allocate memory error. But this
> limit does not take the actual current status into account. ie. Limit is
> not say 95% of total memory etc.

You may be better off with proper RSS limit enforcement patches such as
are present in -ac kernels. Non-overcommit stuff there may also help.


On Tue, Sep 10, 2002 at 01:05:06PM +0530, Srikrishnan Sundararajan wrote:
> I understand that we can allocate quota for hard disk space, there by
> preventing non-root processes from occupying any more disk space beyond the
> quota limit. For example,  we can set quota such that when the
> hard-disk-space is 95% full, only root can occupy further space. Is there a
> similar way to enforce the same for memory usage. This might ensure that
> errant non-root processes cannot keep on allocating memory, thereby can
> prevent the swap from getting full.

RSS limits are on a per-process basis. This kind of workload management
facility has yet to be implemented for Linux.


On Tue, Sep 10, 2002 at 01:05:06PM +0530, Srikrishnan Sundararajan wrote:
> Another thought is can we exclude root processes from  the "badness"
> calculation. This might ensure that at no time a root process is killed by
> oom_kill. Or we can modify this such that as long as a non-root process is
> there, no root processes will be killed by oom_kill.

It's possible, though it hits corner cases of runaway process running with
uid 0 and so doesn't really perform any better than anything else.


On Tue, Sep 10, 2002 at 01:05:06PM +0530, Srikrishnan Sundararajan wrote:
> Also the current oom_kill does not seem to always identify the offending
> process and kill that.  Is there any way of either identifying a specific
> offending process or identify such a user and kill all his processes?

There is no algorithmic method of defining "offending process". I suspect
for these kinds of scenario's Alan's non-overcommit patches would benefit
you more than trying to make overcommit predictable in its worst cases,
especially since that predictability is precisely the tradeoff of
overcommitting memory.


Cheers,
Bill
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: oom_killer - Does not perform when stress-tested (system hangs)

[Srikrishnan Sundararajan]

I  used kernel 2.4.19. This has a few changes in oom_kill.c . It works well
on my intel PC.
ie. Even when I run 500 instances of my memory grabbing program, the
oom_killer is able to kill these errant processes and makes the machine
usable. (No hangs) Hats off to oom_kill!

Thanks,
Srikrishnan




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

oom_killer - Does not perform when stress-tested (system hangs)

[Srikrishnan Sundararajan]

Hi,
I'm trying to run multiple instances of the following program which keeps
on allocating memory , assigns value if malloc was not NULL  and does not
free.
#include <stdlib.h>
int main()
{
   int *p,i;
   while (1)  {
      p = (int *) malloc(1000000 * sizeof(int));
      if(p!=NULL)
          for(i=0;i<1000000;i++)
             p[i]=i;
      sleep(1);
   }
}


When I run say about 5 instances, oom_killer kills one instance of my
program when SwapFree is 0K, goes on to kill each of the other instances in
turn. The machine is slow in response when my program was running but
perfectly usable after that.
When I run 25 or 40 instances, the system hangs. No response. After waiting
for more than 1.5 hours I did a manual reboot (hard-reset). I looked for
/var/log/messages for "Out of Memory: Killed process...", I could find
about 15 entries for the killing of my program's instances, none for others
and there were no entries for more than an hour till I hard-reset the
machine.
I used a PC with Linux -2.4.7-10 (RH 7.2). RAM:128 MB, Swap: 256 MB. I run
as an user and not as root.

Is this expected behavior? Is it the responsibility of the user not to
"fill" the memory? Could oom_killer not take care of such a stress-test?
Should any thing warn the user when swap-space is full?


Srikrishnan


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: oom_killer - Does not perform when stress-tested (system hangs)

[William Lee Irwin III]

On Wed, Aug 07, 2002 at 10:31:21PM +0530, Srikrishnan Sundararajan wrote:
> I used a PC with Linux -2.4.7-10 (RH 7.2). RAM:128 MB, Swap: 256 MB. I run
> as an user and not as root.
> Is this expected behavior? Is it the responsibility of the user not to
> "fill" the memory? Could oom_killer not take care of such a stress-test?
> Should any thing warn the user when swap-space is full?

Can you reproduce this with 2.4.19, 2.4.19-ac, 2.4.19-rmap, or 2.5.30?


Thanks,
Bill
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/