Re: [PATCH] Prevent OOM from killing init

Re: [PATCH] Prevent OOM from killing init

[Rik van Riel]

On 22 Mar 2001, Michael Peddemors wrote:

> Here, Here.. killing qmail on a server who's sole task is running mail
> doesn't seem to make much sense either..

I won't defend the current OOM killing code.

Instead, I'm asking everybody who's unhappy with the
current code to come up with something better.

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] Prevent OOM from killing init

[Guest section DW]

On Fri, Mar 23, 2001 at 04:04:09AM -0300, Rik van Riel wrote:
> On 22 Mar 2001, Michael Peddemors wrote:
> 
> > Here, Here.. killing qmail on a server who's sole task is running mail
> > doesn't seem to make much sense either..
> 
> I won't defend the current OOM killing code.
> 
> Instead, I'm asking everybody who's unhappy with the
> current code to come up with something better.

To a murderer: "Why did you kill that old lady?"
Reply: "I won't defend that deed, but who else should I have killed?"

Andries - getting more and more unhappy with OOM

Mar 23 11:48:49 mette kernel: Out of Memory: Killed process 2019 (emacs).
Mar 23 11:48:49 mette kernel: Out of Memory: Killed process 1407 (emacs).
Mar 23 11:48:50 mette kernel: Out of Memory: Killed process 1495 (emacs).
Mar 23 11:48:50 mette kernel: Out of Memory: Killed process 2800 (rpm).

[yes, that was rpm growing too large, taking a few emacs sessions]
[2.4.2]
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] Prevent OOM from killing init

[Eric W. Biederman]

Guest section DW <dwguest@win.tue.nl> writes:

> On Fri, Mar 23, 2001 at 04:04:09AM -0300, Rik van Riel wrote:
> > On 22 Mar 2001, Michael Peddemors wrote:
> > 
> > > Here, Here.. killing qmail on a server who's sole task is running mail
> > > doesn't seem to make much sense either..
> > 
> > I won't defend the current OOM killing code.
> > 
> > Instead, I'm asking everybody who's unhappy with the
> > current code to come up with something better.
> 
> To a murderer: "Why did you kill that old lady?"
> Reply: "I won't defend that deed, but who else should I have killed?"

> 
> Andries - getting more and more unhappy with OOM
> 
> Mar 23 11:48:49 mette kernel: Out of Memory: Killed process 2019 (emacs).
> Mar 23 11:48:49 mette kernel: Out of Memory: Killed process 1407 (emacs).
> Mar 23 11:48:50 mette kernel: Out of Memory: Killed process 1495 (emacs).
> Mar 23 11:48:50 mette kernel: Out of Memory: Killed process 2800 (rpm).
> 
> [yes, that was rpm growing too large, taking a few emacs sessions]
> [2.4.2]

Let me get this straight you don't have enough swap for your workload?
And you don't have per process limits on root by default?

So you are complaining about the OOM killer?  

Eric
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

General 2.4 impressions (was Re: [PATCH] Prevent OOM from killing init)

[Jeff Garzik]

Personally I think the OOM killer itself is fine.  I think there are
problems elsewhere which are triggering the OOM killer when it should
not be triggered, ie. a leak like Doug Ledford was reporting.

I definitely see heavier page/dcache usage in 2.4 -- but that is to be
expected due to 2.4 changes!  So it is incredibily difficult to quantify
if something is wrong, and if so, where...

My own impressions of 2.4 are that it "feels faster" for my own uses and
it's stable.  The downsides I find are that heavy fs activity seems to
imply increased swapping, which jibes with a guess that the page/dcache
is exceptionally greedy with releasing pages under memory pressure.

</unquantified vague ramble>

-- 
Jeff Garzik       | May you have warm words on a cold evening,
Building 1024     | a full moon on a dark night,
MandrakeSoft      | and a smooth road all the way to your door.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Adding just a pinch of icache/dcache pressure...

[Jan Harkes]

On Fri, Mar 23, 2001 at 10:13:55AM -0500, Jeff Garzik wrote:
> Personally I think the OOM killer itself is fine.  I think there are
> problems elsewhere which are triggering the OOM killer when it should
> not be triggered, ie. a leak like Doug Ledford was reporting.
> 
> I definitely see heavier page/dcache usage in 2.4 -- but that is to be
> expected due to 2.4 changes!  So it is incredibily difficult to quantify
> if something is wrong, and if so, where...
> 
> My own impressions of 2.4 are that it "feels faster" for my own uses and
> it's stable.  The downsides I find are that heavy fs activity seems to
> imply increased swapping, which jibes with a guess that the page/dcache
> is exceptionally greedy with releasing pages under memory pressure.
> 
> </unquantified vague ramble>

Like I said earlier, I should stop theorizing and write the code. Here
is a teeny little patch that adds a bit of pressure to the inode and
dentry slabcaches during inactive shortage.

On the 512MB desktop without the change, the inode+dentry slabs
typically used up about 300MB after running my normal day-to-day
workload for about 24 hours. Now, the inode+dentry slabs are using
only 90MB.

As there is more memory available for the buffer and page caches, kswapd
seems to have less trouble keeping up with my typical workload.


btw. There definitely is a network receive buffer leak somewhere in
either the 3c905C path or higher up in the network layers (2.4.0 or
2.4.1). The normal path does not leak anything.

I was seeing it only for a couple of days when there was a failing
switch that must have randomly corrupted packets. The switch got
replaced and the leakage disappeared, so I went back into a non-ikd
kernel and stopped looking for the problem.

Jan


=================
--- linux/fs/inode.c.orig	Thu Mar 22 13:20:55 2001
+++ linux/fs/inode.c	Thu Mar 22 14:00:10 2001
@@ -270,19 +270,6 @@
 	spin_unlock(&inode_lock);
 }
 
-/*
- * Called with the spinlock already held..
- */
-static void sync_all_inodes(void)
-{
-	struct super_block * sb = sb_entry(super_blocks.next);
-	for (; sb != sb_entry(&super_blocks); sb = sb_entry(sb->s_list.next)) {
-		if (!sb->s_dev)
-			continue;
-		sync_list(&sb->s_dirty);
-	}
-}
-
 /**
  *	write_inode_now	-	write an inode to disk
  *	@inode: inode to write to disk
@@ -507,8 +494,6 @@
 	struct inode * inode;
 
 	spin_lock(&inode_lock);
-	/* go simple and safe syncing everything before starting */
-	sync_all_inodes();
 
 	entry = inode_unused.prev;
 	while (entry != &inode_unused)
@@ -554,6 +539,9 @@
 
 	if (priority)
 		count = inodes_stat.nr_unused / priority;
+
+	if (priority < 6)
+		sync_inodes(0);
 
 	prune_icache(count);
 	kmem_cache_shrink(inode_cachep);
--- linux/mm/vmscan.c.orig	Thu Mar 22 14:00:41 2001
+++ linux/mm/vmscan.c	Thu Mar 22 14:35:26 2001
@@ -845,9 +845,11 @@
 	 * reclaim unused slab cache if memory is low.
 	 */
 	if (free_shortage()) {
+		shrink_dcache_memory(5, gfp_mask);
+		shrink_icache_memory(5, gfp_mask);
+	} else {
 		shrink_dcache_memory(DEF_PRIORITY, gfp_mask);
 		shrink_icache_memory(DEF_PRIORITY, gfp_mask);
-	} else {
 		/*
 		 * Illogical, but true. At least for now.
 		 *
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: Adding just a pinch of icache/dcache pressure...

[Andi Kleen]

On Fri, Mar 23, 2001 at 05:10:56PM +0100, Jan Harkes wrote:
> btw. There definitely is a network receive buffer leak somewhere in
> either the 3c905C path or higher up in the network layers (2.4.0 or
> 2.4.1). The normal path does not leak anything.


What do you mean with "normal path" ? 

And are you sure it was a leak? TCP can buffer quite a bit of skbs, but it 
should be bounded based on the number of sockets. 


-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] Prevent OOM from killing init

[Guest section DW]

On Fri, Mar 23, 2001 at 07:50:25AM -0700, Eric W. Biederman wrote:

> > Mar 23 11:48:49 mette kernel: Out of Memory: Killed process 2019 (emacs).
> > Mar 23 11:48:49 mette kernel: Out of Memory: Killed process 1407 (emacs).
> > Mar 23 11:48:50 mette kernel: Out of Memory: Killed process 1495 (emacs).
> > Mar 23 11:48:50 mette kernel: Out of Memory: Killed process 2800 (rpm).
> > 
> > [yes, that was rpm growing too large, taking a few emacs sessions]
> > [2.4.2]
> 
> Let me get this straight you don't have enough swap for your workload?
> And you don't have per process limits on root by default?
> 
> So you are complaining about the OOM killer?  

I should not react - your questions are phrased rhetorically.

But yes, I am complaining because Linux by default is unreliable.
I strongly prefer a system that is reliable by default,
and I'll leave it to others to run it in an unreliable mode.

Andries
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] Prevent OOM from killing init

[Paul Jakma]

On Fri, 23 Mar 2001, Guest section DW wrote:

> But yes, I am complaining because Linux by default is unreliable.

no, your distribution is unreliable by default.

> I strongly prefer a system that is reliable by default,
> and I'll leave it to others to run it in an unreliable mode.

currently, setting sensible user limits on my machines means i never
get a hosed machine due to OOM. These limits are easy to set via
pam_limits. (not perfect though, i think its session specific..)

granted, if the machine hasn't been setup with user limits, then linux
doesn't deal at all well with OOM, so this should be fixed. but it can
easily be argued that admin error in not configuring limits is the
main cause for OOM.

> Andries

regards,

--paulj

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] Prevent OOM from killing init

[Jesse Pollard]

On Fri, 23 Mar 2001, Paul Jakma wrote:
>On Fri, 23 Mar 2001, Guest section DW wrote:
>
>> But yes, I am complaining because Linux by default is unreliable.
>
>no, your distribution is unreliable by default.
>
>> I strongly prefer a system that is reliable by default,
>> and I'll leave it to others to run it in an unreliable mode.
>
>currently, setting sensible user limits on my machines means i never
>get a hosed machine due to OOM. These limits are easy to set via
>pam_limits. (not perfect though, i think its session specific..)

Process specific. Each forked process gets the same limits. You get OOM
as soon as all processes together use more than the system capacity.

>granted, if the machine hasn't been setup with user limits, then linux
>doesn't deal at all well with OOM, so this should be fixed. but it can
>easily be argued that admin error in not configuring limits is the
>main cause for OOM.

Admin has no real control is the problem. Limits are only good for one
process. As soon as that process forks one other process then the
useage limit is twice the limit established.

>> Andries
>
>regards,
>
>--paulj

-- 
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: jesse@cats-chateau.net

Any opinions expressed are solely my own.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

Re: [PATCH] Prevent OOM from killing init

[Eric W. Biederman]

Guest section DW <dwguest@win.tue.nl> writes:

> On Fri, Mar 23, 2001 at 07:50:25AM -0700, Eric W. Biederman wrote:
> 
> > > Mar 23 11:48:49 mette kernel: Out of Memory: Killed process 2019 (emacs).
> > > Mar 23 11:48:49 mette kernel: Out of Memory: Killed process 1407 (emacs).
> > > Mar 23 11:48:50 mette kernel: Out of Memory: Killed process 1495 (emacs).
> > > Mar 23 11:48:50 mette kernel: Out of Memory: Killed process 2800 (rpm).
> > > 
> > > [yes, that was rpm growing too large, taking a few emacs sessions]
> > > [2.4.2]
> > 
> > Let me get this straight you don't have enough swap for your workload?
> > And you don't have per process limits on root by default?
> > 
> > So you are complaining about the OOM killer?  
> 
> I should not react - your questions are phrased rhetorically.

To some extent I was also very puzzled by your complaint.

You have setup a system that by your definition unreliably and then
you complain it is unreliable.

> 
> But yes, I am complaining because Linux by default is unreliable.
> I strongly prefer a system that is reliable by default,
> and I'll leave it to others to run it in an unreliable mode.

Now all I know the system didn't have enough resources to do what
you asked to it do and it failed.  That sounds reliable to me.  

Obviously you were suprised at how the system failed.  Given
that unix has been doing this kind of thing for decades, you obviously
missed how the unix malloc overcommited memory.

Does you application trap sigsegv on a different stack so you can
catch stack growth failure?  And how does your app handle this case?

Having a no over commit kernel option would help.  

A cheap workaround is to call mlock_all(MCL_FUTRE...).  Then you are
garantteed you will always have ram locked into memory for your
program.   This assumes you have enough ram for your program.

Eric

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/