[PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Marcelo Tosatti]

Hi,

As you know the OOM is very problematic in 2.6 right now - so I went
to investigate it. 

Currently the oom killer is invoked from the task reclaim 
code (try_to_free_pages), which IMO is fundamentally broken, 
because its non deterministic - the chance the OOM killer 
will be triggered increases as the number of tasks inside 
reclaiming increases. And kswapd is freeing pages in parallel, 
which is completly ignored by this approach.

In my opinion the correct approach is to trigger the OOM killer 
when kswapd is unable to free pages. Once that is done, the number 
of tasks inside page reclaim is irrelevant. 

So the following patch moves the out_of_memory() call to 
balance_pgdat(), and makes it dependant on success reclaiming pages 
when work has actually been done, and on priority reaching zero.

It also removes the "If it's been a long time since last failure dont 
OOM kill" logic which for me just tries to paper over a bigger issue. 

Relying on this information (kswapd failure after DEF_PRIORITY passes) 
to trigger the OOM killer seems to be very reliable - it needs some 
more testing though.

With this in place I can't see spurious OOM kills - just need to guarantee
that it reliably OOM kills when we are really out of memory.

While doing this, I noticed that kswapd will happily go to sleep 
if all zones have all_unreclaimable set. I bet this is the reason 
for the page allocation failures we are seeing. So the patch 
also makes balance_pgdat() NOT return and go to "loop_again" 
instead in case of page shortage - even if all_unreclaimable is set.

Basically the "loop_again" logic IS NOT WORKING! 

Comments?

My wife is almost killing me, its Friday night and I've been telling her 
"just another minute" for hours. Have to run.

diff -Nur --show-c-function --exclude='*.orig' linux-2.6.10-rc1-mm2.orig/mm/oom_kill.c linux-2.6.10-rc1-mm2/mm/oom_kill.c
--- linux-2.6.10-rc1-mm2.orig/mm/oom_kill.c	2004-11-04 22:50:50.000000000 -0200
+++ linux-2.6.10-rc1-mm2/mm/oom_kill.c	2004-11-05 18:33:29.918459072 -0200
@@ -240,23 +240,23 @@ void out_of_memory(int gfp_mask)
 	 * If it's been a long time since last failure,
 	 * we're not oom.
 	 */
-	if (since > 5*HZ)
-		goto reset;
+	//if (since > 5*HZ)
+	//	goto reset;
 
 	/*
 	 * If we haven't tried for at least one second,
 	 * we're not really oom.
 	 */
-	since = now - first;
-	if (since < HZ)
-		goto out_unlock;
+	//since = now - first;
+	//if (since < HZ)
+	//	goto out_unlock;
 
 	/*
 	 * If we have gotten only a few failures,
 	 * we're not really oom. 
 	 */
-	if (++count < 10)
-		goto out_unlock;
+//	if (++count < 10)
+//		goto out_unlock;
 
 	/*
 	 * If we just killed a process, wait a while
diff -Nur --show-c-function --exclude='*.orig' linux-2.6.10-rc1-mm2.orig/mm/vmscan.c linux-2.6.10-rc1-mm2/mm/vmscan.c
--- linux-2.6.10-rc1-mm2.orig/mm/vmscan.c	2004-11-04 22:50:50.000000000 -0200
+++ linux-2.6.10-rc1-mm2/mm/vmscan.c	2004-11-05 19:53:35.915836416 -0200
@@ -952,8 +952,6 @@ int try_to_free_pages(struct zone **zone
 		if (sc.nr_scanned && priority < DEF_PRIORITY - 2)
 			blk_congestion_wait(WRITE, HZ/10);
 	}
-	if ((gfp_mask & __GFP_FS) && !(gfp_mask & __GFP_NORETRY))
-		out_of_memory(gfp_mask);
 out:
 	for (i = 0; zones[i] != 0; i++) {
 		struct zone *zone = zones[i];
@@ -997,13 +995,15 @@ static int balance_pgdat(pg_data_t *pgda
 	int all_zones_ok;
 	int priority;
 	int i;
-	int total_scanned, total_reclaimed;
+	int total_scanned, total_reclaimed, worked;
 	struct reclaim_state *reclaim_state = current->reclaim_state;
 	struct scan_control sc;
 
+
 loop_again:
 	total_scanned = 0;
 	total_reclaimed = 0;
+	worked = 0;
 	sc.gfp_mask = GFP_KERNEL;
 	sc.may_writepage = 0;
 	sc.nr_mapped = read_page_state(nr_mapped);
@@ -1033,6 +1033,10 @@ loop_again:
 				if (zone->present_pages == 0)
 					continue;
 
+				if (!zone_watermark_ok(zone, order, 
+						zone->pages_high, 0, 0, 0))
+					all_zones_ok = 0;
+
 				if (zone->all_unreclaimable &&
 						priority != DEF_PRIORITY)
 					continue;
@@ -1072,6 +1076,9 @@ scan:
 			if (zone->all_unreclaimable && priority != DEF_PRIORITY)
 				continue;
 
+			if (priority == 0)
+				worked = 1;
+
 			if (nr_pages == 0) {	/* Not software suspend */
 				if (!zone_watermark_ok(zone, order,
 						zone->pages_high, end_zone, 0, 0))
@@ -1129,6 +1136,9 @@ out:
 		zone->prev_priority = zone->temp_priority;
 	}
 	if (!all_zones_ok) {
+		if (priority == 0 && !total_reclaimed && worked)
+			out_of_memory(GFP_KERNEL);
+
 		cond_resched();
 		goto loop_again;
 	}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Jesse Barnes]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

On Friday, November 05, 2004 12:01 pm, Marcelo Tosatti wrote:
> Hi,
>
> As you know the OOM is very problematic in 2.6 right now - so I went
> to investigate it.
>
> Currently the oom killer is invoked from the task reclaim
> code (try_to_free_pages), which IMO is fundamentally broken,
> because its non deterministic - the chance the OOM killer
> will be triggered increases as the number of tasks inside
> reclaiming increases. And kswapd is freeing pages in parallel,
> which is completly ignored by this approach.
>
> In my opinion the correct approach is to trigger the OOM killer
> when kswapd is unable to free pages. Once that is done, the number
> of tasks inside page reclaim is irrelevant.

That makes sense.

> With this in place I can't see spurious OOM kills - just need to guarantee
> that it reliably OOM kills when we are really out of memory.

That's good.  I can test it on a large machine (hopefully next week).

> Comments?

Sounds good, though we may want to do a couple of more things, we shouldn't 
kill root tasks quite as easily and we should avoid zombies since they may be 
large apps in the process of exiting, and killing them would be bad (iirc 
it'll cause a panic).

Thanks,
Jesse

[-- Attachment #2: oom-fixes.patch --]
[-- Type: text/x-diff, Size: 623 bytes --]

===== mm/oom_kill.c 1.30 vs edited =====
--- 1.30/mm/oom_kill.c	2004-10-13 21:08:29 -07:00
+++ edited/mm/oom_kill.c	2004-11-05 15:32:36 -08:00
@@ -88,7 +88,7 @@
 	 */
 	if (cap_t(p->cap_effective) & CAP_TO_MASK(CAP_SYS_ADMIN) ||
 				p->uid == 0 || p->euid == 0)
-		points /= 4;
+		points /= 10;
 
 	/*
 	 * We don't want to kill a process with direct hardware access.
@@ -120,7 +120,7 @@
 
 	do_posix_clock_monotonic_gettime(&uptime);
 	do_each_thread(g, p)
-		if (p->pid) {
+		if (p->pid && !(p->state & TASK_ZOMBIE)) {
 			unsigned long points = badness(p, uptime.tv_sec);
 			if (points > maxpoints) {
 				chosen = p;

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Thomas Gleixner]

On Fri, 2004-11-05 at 15:32 -0800, Jesse Barnes wrote:
> On Friday, November 05, 2004 12:01 pm, Marcelo Tosatti wrote:
> > Comments?
> 
> Sounds good, though we may want to do a couple of more things, we shouldn't 
> kill root tasks quite as easily and we should avoid zombies since they may be 
> large apps in the process of exiting, and killing them would be bad (iirc 
> it'll cause a panic).
> 

Yep, it makes sense, but it still does not fix the selection problem,
where e.g. sshd is killed while a out of control forking server floods
the machine with child processes. 

Patch to address this:
 http://marc.theaimsgroup.com/?l=linux-kernel&m=109922680000746&w=2

tglx






--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Andrea Arcangeli]

On Fri, Nov 05, 2004 at 03:32:50PM -0800, Jesse Barnes wrote:
> On Friday, November 05, 2004 12:01 pm, Marcelo Tosatti wrote:
> > In my opinion the correct approach is to trigger the OOM killer
> > when kswapd is unable to free pages. Once that is done, the number
> > of tasks inside page reclaim is irrelevant.
> 
> That makes sense.

I don't like it, kswapd may fail balancing because there's a GFP_DMA
allocation that eat the last dma page, but we should not kill tasks if
we fail to balance in kswapd, we should kill tasks only when no fail
path exists (i.e. only during page faults, everything else in the kernel
has a fail path and it should never trigger oom).

If you move it in kswapd there's no way to prevent oom-killing from a
syscall allocation (I guess even right now it would go wrong in this
sense, but at least right now it's more fixable). I want to move the oom
kill outside the alloc_page paths. The oom killing is all about the page
faults not having a fail path, and in turn the oom killing should be
moved in the page fault code, not in the allocator. Everything else
should keep returning -ENOMEM to the caller.

So to me moving the oom killer into kswapd looks a regression.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Nick Piggin]

Andrea Arcangeli wrote:

>On Fri, Nov 05, 2004 at 03:32:50PM -0800, Jesse Barnes wrote:
>
>>On Friday, November 05, 2004 12:01 pm, Marcelo Tosatti wrote:
>>
>>>In my opinion the correct approach is to trigger the OOM killer
>>>when kswapd is unable to free pages. Once that is done, the number
>>>of tasks inside page reclaim is irrelevant.
>>>
>>That makes sense.
>>
>
>I don't like it, kswapd may fail balancing because there's a GFP_DMA
>allocation that eat the last dma page, but we should not kill tasks if
>we fail to balance in kswapd, we should kill tasks only when no fail
>path exists (i.e. only during page faults, everything else in the kernel
>has a fail path and it should never trigger oom).
>
>If you move it in kswapd there's no way to prevent oom-killing from a
>syscall allocation (I guess even right now it would go wrong in this
>sense, but at least right now it's more fixable). I want to move the oom
>kill outside the alloc_page paths. The oom killing is all about the page
>faults not having a fail path, and in turn the oom killing should be
>moved in the page fault code, not in the allocator. Everything else
>should keep returning -ENOMEM to the caller.
>
>

Probably a good idea. OTOH, some kernel allocations might really
need to be performed and have no failure path. For example __GFP_REPEAT.

I think maybe __GFP_REPEAT allocations at least should be able to
cause an OOM. Not sure though.

>So to me moving the oom killer into kswapd looks a regression.
>
>
>

Also, I think it would do the wrong thing on NUMA machines because
that has a per-node kswapd.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Jesse Barnes]

On Friday, November 05, 2004 5:26 pm, Nick Piggin wrote:
> >If you move it in kswapd there's no way to prevent oom-killing from a
> >syscall allocation (I guess even right now it would go wrong in this
> >sense, but at least right now it's more fixable). I want to move the oom
> >kill outside the alloc_page paths. The oom killing is all about the page
> >faults not having a fail path, and in turn the oom killing should be
> >moved in the page fault code, not in the allocator. Everything else
> >should keep returning -ENOMEM to the caller.
>
> Probably a good idea. OTOH, some kernel allocations might really
> need to be performed and have no failure path. For example __GFP_REPEAT.

Ah, I see what you're saying, yes, that makes even more sense :)

> I think maybe __GFP_REPEAT allocations at least should be able to
> cause an OOM. Not sure though.
>
> >So to me moving the oom killer into kswapd looks a regression.
>
> Also, I think it would do the wrong thing on NUMA machines because
> that has a per-node kswapd.

Yep, Andrea's explaination is clear, I just had to read it a few times.  
Anyway, the fixes I posted are still necessary I think.

Thanks,
Jesse
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Andrea Arcangeli]

On Sat, Nov 06, 2004 at 12:26:57PM +1100, Nick Piggin wrote:
> need to be performed and have no failure path. For example __GFP_REPEAT.

all allocations should have a failure path to avoid deadlocks. But in
the meantime __GFP_REPEAT is at least localizing the problematic places ;)

> I think maybe __GFP_REPEAT allocations at least should be able to
> cause an OOM. Not sure though.

probably it should because this is also a case where no fail path exists.

My point was only that when a fail path exists, it's more reliable not
to invoke the oom killer and let userspace handle the failure.

> Also, I think it would do the wrong thing on NUMA machines because
> that has a per-node kswapd.

yep.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Hugh Dickins]

On Sat, 6 Nov 2004, Andrea Arcangeli wrote:
> 
> all allocations should have a failure path to avoid deadlocks. But in
> the meantime __GFP_REPEAT is at least localizing the problematic places ;)

Problematic, yes: don't overlook that GFP_REPEAT and GFP_NOFAIL _can_
fail, returning NULL: when the process is being OOM-killed (PF_MEMDIE).

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Nick Piggin]

Hugh Dickins wrote:

>On Sat, 6 Nov 2004, Andrea Arcangeli wrote:
>
>>all allocations should have a failure path to avoid deadlocks. But in
>>the meantime __GFP_REPEAT is at least localizing the problematic places ;)
>>
>
>Problematic, yes: don't overlook that GFP_REPEAT and GFP_NOFAIL _can_
>fail, returning NULL: when the process is being OOM-killed (PF_MEMDIE).
>
>

Yeah right you are. I think NOFAIL is a bug and should really not fail.
It looks like it is only used in fs/jbd/*, and things will crash if it
fails. Maybe they're only called from the kjournald threads and can't
be OOM killed, but that is still a pretty subtle dependancy.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Andrea Arcangeli]

On Sat, Nov 06, 2004 at 09:53:00PM +1100, Nick Piggin wrote:
> Yeah right you are. I think NOFAIL is a bug and should really not fail.

agreed.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Andrea Arcangeli]

On Sat, Nov 06, 2004 at 09:47:56AM +0000, Hugh Dickins wrote:
> Problematic, yes: don't overlook that GFP_REPEAT and GFP_NOFAIL _can_
> fail, returning NULL: when the process is being OOM-killed (PF_MEMDIE).

that looks weird, why that? The oom killer must be robust against a task
not going anyway regardless of this (task can be stuck in nfs or
similar). If a fail path ever existed, __GFP_NOFAIL should not have been
used in the first place. I don't see many valid excuses to use
__GFP_NOFAIL if we can return NULL without the caller running into an
infinite loop.

btw, PF_MEMDIE has always been racy in the way it's being set, so it can
corrupt the p->flags, but the race window is very small to trigger it
(and even if it triggers, it probably wouldn't be fatal). That's why I
don't use PF_MEMDIE in 2.4-aa.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Hugh Dickins]

On Sat, 6 Nov 2004, Andrea Arcangeli wrote:
> On Sat, Nov 06, 2004 at 09:47:56AM +0000, Hugh Dickins wrote:
> > Problematic, yes: don't overlook that GFP_REPEAT and GFP_NOFAIL _can_
> > fail, returning NULL: when the process is being OOM-killed (PF_MEMDIE).
> 
> that looks weird, why that? The oom killer must be robust against a task
> not going anyway regardless of this (task can be stuck in nfs or
> similar).

Oh, sure, it is, that's not the problem.

> If a fail path ever existed, __GFP_NOFAIL should not have been
> used in the first place. I don't see many valid excuses to use
> __GFP_NOFAIL if we can return NULL without the caller running into an
> infinite loop.

I took exception to the misleadingness of the name GFP_NOFAIL, and did
send Andrew a patch to remove it once upon a time, but he didn't bite.

Your view, that it's better to hang repeating indefinitely than ever
return a NULL when caller said not to, is probably the better view.

> btw, PF_MEMDIE has always been racy in the way it's being set, so it can
> corrupt the p->flags, but the race window is very small to trigger it
> (and even if it triggers, it probably wouldn't be fatal). That's why I
> don't use PF_MEMDIE in 2.4-aa.

I expect so, yes, the PF_ flags don't have proper locking.  Those
places which set or clear PF_MEMALLOC are more likely to hit races,
but last time I went there I don't think there was a real serious problem.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[William Lee Irwin III]

On Sat, 6 Nov 2004, Andrea Arcangeli wrote:
>> btw, PF_MEMDIE has always been racy in the way it's being set, so it can
>> corrupt the p->flags, but the race window is very small to trigger it
>> (and even if it triggers, it probably wouldn't be fatal). That's why I
>> don't use PF_MEMDIE in 2.4-aa.

On Sat, Nov 06, 2004 at 04:21:33PM +0000, Hugh Dickins wrote:
> I expect so, yes, the PF_ flags don't have proper locking.  Those
> places which set or clear PF_MEMALLOC are more likely to hit races,
> but last time I went there I don't think there was a real serious problem.

I posted a testcase that triggers a panic with the PF_MEMALLOC race.


-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Nikita Danilov]

Andrea Arcangeli writes:
 > On Sat, Nov 06, 2004 at 12:26:57PM +1100, Nick Piggin wrote:
 > > need to be performed and have no failure path. For example __GFP_REPEAT.
 > 
 > all allocations should have a failure path to avoid deadlocks. But in

This is not currently possible for a complex operation that allocates
multiple pages and has always complete as a whole.

We need page-reservation API of some sort. There were several attempts
to introduce this, but none get into mainline.

Nikita.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Andrea Arcangeli]

On Sat, Nov 06, 2004 at 02:37:05PM +0300, Nikita Danilov wrote:
> We need page-reservation API of some sort. There were several attempts
> to introduce this, but none get into mainline.

they're already in under the name of mempools

I'm perfectly aware the fs tends to be the less correct places in terms
of allocations, and luckily it's not an heavy memory user, so I still
have to see a deadlock in getblk or create_buffers or similar. It's
mostly a correctness issue (math proof it can't deadlock, right now it
can if more tasks all get stuck in getblk at the same time during a hard
oom condition etc..).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Nikita Danilov]

Andrea Arcangeli writes:
 > On Sat, Nov 06, 2004 at 02:37:05PM +0300, Nikita Danilov wrote:
 > > We need page-reservation API of some sort. There were several attempts
 > > to introduce this, but none get into mainline.
 > 
 > they're already in under the name of mempools

I am talking about slightly different thing. Think of some operation
that calls find_or_create_page(). find_or_create_page() doesn't know
about memory reserved in mempools, it uses alloc_page() directly. If one
wants to guarantee that compound operation has enough memory to
complete, memory should be reserved at the lowest level---in the page
allocator.

 > 
 > I'm perfectly aware the fs tends to be the less correct places in terms
 > of allocations, and luckily it's not an heavy memory user, so I still

Either you are kidding, or we are facing very different workloads. In
the world of file-system development, file-system is (not surprisingly)
single largest memory consumer.

 > have to see a deadlock in getblk or create_buffers or similar. It's
 > mostly a correctness issue (math proof it can't deadlock, right now it
 > can if more tasks all get stuck in getblk at the same time during a hard
 > oom condition etc..).

Add here mmap that can dirty all physical memory behind your back, and
delayed disk block allocation that forces ->writepage() to allocate
potentially huge extent when memory is already tight and hope of having
a proof becomes quite remote.

Nikita.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Andrea Arcangeli]

On Sat, Nov 06, 2004 at 07:54:12PM +0300, Nikita Danilov wrote:
> Andrea Arcangeli writes:
>  > On Sat, Nov 06, 2004 at 02:37:05PM +0300, Nikita Danilov wrote:
>  > > We need page-reservation API of some sort. There were several attempts
>  > > to introduce this, but none get into mainline.
>  > 
>  > they're already in under the name of mempools
> 
> I am talking about slightly different thing. Think of some operation
> that calls find_or_create_page(). find_or_create_page() doesn't know
> about memory reserved in mempools, it uses alloc_page() directly. If one
> wants to guarantee that compound operation has enough memory to
> complete, memory should be reserved at the lowest level---in the page
> allocator.

the page allocator reserve only memory in order to swapout, that's
PF_MEMALLOC.

For other purposes not related to swapping (which is not a deterministic
thing, given there can be multiple layers of I/O and fs operations to
do), you should use mempool and change find_or_create_page to get your
reserved page as parameter.

>  > I'm perfectly aware the fs tends to be the less correct places in terms
>  > of allocations, and luckily it's not an heavy memory user, so I still
> 
> Either you are kidding, or we are facing very different workloads. In
> the world of file-system development, file-system is (not surprisingly)
> single largest memory consumer.

when the machine runs oom the fs allocations means nothing. when the
machine runs oom is because somebody entered the malloc loop or
something like that. all allocations come from page faults.

Try yourself to run your box oom with getblk allocations. Only then
you'll run into the deadlock.

You've to keep in mind an oom condition happens once in a while, and
when it happens the userspace memory allocation load is huge compared to
any fs operation.

>  > have to see a deadlock in getblk or create_buffers or similar. It's
>  > mostly a correctness issue (math proof it can't deadlock, right now it
>  > can if more tasks all get stuck in getblk at the same time during a hard
>  > oom condition etc..).
> 
> Add here mmap that can dirty all physical memory behind your back, and
> delayed disk block allocation that forces ->writepage() to allocate
> potentially huge extent when memory is already tight and hope of having
> a proof becomes quite remote.

that's the PF_MEMALLOC path. A reservation already exists, or it would
never work since 2.2. PF_MEMALLOC and the min/2 watermark are meant to
allow writepage to allocate ram. however the amount reserved is limited,
so it's not perfect. The only way to make it perfect I believe is to
reserve the stuff inside the fs with mempools as described above.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Nikita Danilov]

Andrea Arcangeli writes:
 > On Sat, Nov 06, 2004 at 07:54:12PM +0300, Nikita Danilov wrote:
 > > Andrea Arcangeli writes:
 > >  > On Sat, Nov 06, 2004 at 02:37:05PM +0300, Nikita Danilov wrote:
 > >  > > We need page-reservation API of some sort. There were several attempts
 > >  > > to introduce this, but none get into mainline.
 > >  > 
 > >  > they're already in under the name of mempools
 > > 
 > > I am talking about slightly different thing. Think of some operation
 > > that calls find_or_create_page(). find_or_create_page() doesn't know
 > > about memory reserved in mempools, it uses alloc_page() directly. If one
 > > wants to guarantee that compound operation has enough memory to
 > > complete, memory should be reserved at the lowest level---in the page
 > > allocator.
 > 
 > the page allocator reserve only memory in order to swapout, that's
 > PF_MEMALLOC.
 > 
 > For other purposes not related to swapping (which is not a deterministic
 > thing, given there can be multiple layers of I/O and fs operations to
 > do), you should use mempool and change find_or_create_page to get your
 > reserved page as parameter.

This means breaking all layering and passing mempool pointer all the way
down to the lowest layer allocators (like bio and drivers). The only
practical way to do this, is to put mempool pointer into current
task_struct. At which point it's no different from having per-thread
list of pages that __alloc_pages() looks into before falling back to
per-cpu page-sets and buddy. _Except_ in the latter case, reservation is
handled transparently in __alloc_pages() and code shouldn't be adjusted
to check for mempool in zillion of places.

 > 
 > >  > I'm perfectly aware the fs tends to be the less correct places in terms
 > >  > of allocations, and luckily it's not an heavy memory user, so I still
 > > 
 > > Either you are kidding, or we are facing very different workloads. In
 > > the world of file-system development, file-system is (not surprisingly)
 > > single largest memory consumer.
 > 
 > when the machine runs oom the fs allocations means nothing. when the
 > machine runs oom is because somebody entered the malloc loop or
 > something like that. all allocations come from page faults.
 > 
 > Try yourself to run your box oom with getblk allocations. Only then
 > you'll run into the deadlock.
 > 
 > You've to keep in mind an oom condition happens once in a while, and
 > when it happens the userspace memory allocation load is huge compared to
 > any fs operation.

I think you are confusing "file system" and "ext2". I definitely know
from experience that with some file system types, system can be oommed
without any significant user-level allocation activity. Now, one can say
that either such file-systems are broken, or Linux MM lacks support for
features (like reservation) they need.

 > 
 > >  > have to see a deadlock in getblk or create_buffers or similar. It's
 > >  > mostly a correctness issue (math proof it can't deadlock, right now it
 > >  > can if more tasks all get stuck in getblk at the same time during a hard
 > >  > oom condition etc..).
 > > 
 > > Add here mmap that can dirty all physical memory behind your back, and
 > > delayed disk block allocation that forces ->writepage() to allocate
 > > potentially huge extent when memory is already tight and hope of having
 > > a proof becomes quite remote.
 > 
 > that's the PF_MEMALLOC path. A reservation already exists, or it would
 > never work since 2.2. PF_MEMALLOC and the min/2 watermark are meant to
 > allow writepage to allocate ram. however the amount reserved is limited,

low-mem watermark is mostly useless in the face of direct reclaim, when
unbounded number of threads enter try_to_free_pages() and call
->writepage() simultaneously.

 > so it's not perfect. The only way to make it perfect I believe is to
 > reserve the stuff inside the fs with mempools as described above.

I don't see what advantages mempools have over page reservation handled
directly by page allocator, like in

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/broken-out/reiser4-perthread-pages.patch

Nikita.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Andrea Arcangeli]

On Sat, Nov 06, 2004 at 10:24:58PM +0300, Nikita Danilov wrote:
> This means breaking all layering and passing mempool pointer all the way
> down to the lowest layer allocators (like bio and drivers). The only

bio and drivers already have their own mempools.  the blkdev layer is
guaranteed to succeed and it can only try GFP_NOIO allocations (if those
fails it'll fallback in the reserved mempool).

> practical way to do this, is to put mempool pointer into current
> task_struct. At which point it's no different from having per-thread
> list of pages that __alloc_pages() looks into before falling back to
> per-cpu page-sets and buddy. _Except_ in the latter case, reservation is
> handled transparently in __alloc_pages() and code shouldn't be adjusted
> to check for mempool in zillion of places.

that's sure reasonable to avoid changing lots of code.

> I think you are confusing "file system" and "ext2". I definitely know
> from experience that with some file system types, system can be oommed
> without any significant user-level allocation activity. Now, one can say
> that either such file-systems are broken, or Linux MM lacks support for
> features (like reservation) they need.

the latter is true, I agree.

>  > that's the PF_MEMALLOC path. A reservation already exists, or it would
>  > never work since 2.2. PF_MEMALLOC and the min/2 watermark are meant to
>  > allow writepage to allocate ram. however the amount reserved is limited,
> 
> low-mem watermark is mostly useless in the face of direct reclaim, when
> unbounded number of threads enter try_to_free_pages() and call
> ->writepage() simultaneously.

agreed.

>  > so it's not perfect. The only way to make it perfect I believe is to
>  > reserve the stuff inside the fs with mempools as described above.
> 
> I don't see what advantages mempools have over page reservation handled
> directly by page allocator, like in
> 
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/broken-out/reiser4-perthread-pages.patch

guess what, that patch is running in my kernel right now. However I
believe this approch is very wasteful. I agree it'll work right, but
you're wasting loads of ram and you're as well less efficient.

The efficient fix for your problem, is to have a global pool, protected
by a global semaphore (definitely not per-thread), so that when you hit
oom (and when you hit true oom the last thing you can care about is
paralleism or the scalability on such a global semaphore), the VM will
trasparently take the semaphore and start using the pool. This will
still require you to mark the start and end of your critical section
like this:

reiserf4_writepage()
{
	enable_reserved_pages_pool();

	find_or_create_page()
	journal something
	getblk
	biowhatever

	disable_reserved_pages_pool();
}

disable_reserved_pages_pool has to check a per-thread flag that the VM
will set if it has used the reserved pool and taken the semaphore, but
by that time the I/O can be guaranteed to complete and the memory will
be guaranteed to be unlocked eventually when the bio I/O completes. So
you can freely alloc_pages to refill the pool inside
disable_reserved_pages_pool and then drop the semaphore.
enable_reserved_pages_pool is only needed to set a per-thread flag to
tell the VM it's allowed to fallback in the global pool by blocking in
the global semaphore if the box is oom (instead of returning NULL).

in disable_reserved_pages_pool you'll also have to clear the pre-thread
flag before calling alloc_pages again to avoid deadlock on the semaphore
if another oom condition happens of course.

then you need an create_reserved_pages_pool(nr_pages) while you mount the
fs, and destroy_unreserve_pages_pool(nr_pages) when you unmont it. where
many different users (i.e. different fs) will be allowed to reserve a
different size for the global pool. They all will share the same pool,
you've only need to track each user nr_pages to know which is the max
reservation you need.

That's still enterely transparent, it'll work in the thread context
thanks to the global semaphore, but it'll avoid the waste of ram where
every different task has to pin the ram into the task before starting
the writepage I/O.

I mean, I understand the only point of the perthread-pages patch is
deadlock avoidance during OOM. So you definitely don't need a per-thread
reservation, the global pool methods I described above should be more
than enough and they'll save ram and make your system faster as well.

I agree PF_MEMALLOC has nothing to do with this.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Marcelo Tosatti]

On Sat, Nov 06, 2004 at 12:26:57PM +1100, Nick Piggin wrote:
> 
> 
> Andrea Arcangeli wrote:
> 
> >On Fri, Nov 05, 2004 at 03:32:50PM -0800, Jesse Barnes wrote:
> >
> >>On Friday, November 05, 2004 12:01 pm, Marcelo Tosatti wrote:
> >>
> >>>In my opinion the correct approach is to trigger the OOM killer
> >>>when kswapd is unable to free pages. Once that is done, the number
> >>>of tasks inside page reclaim is irrelevant.
> >>>
> >>That makes sense.
> >>
> >
> >I don't like it, kswapd may fail balancing because there's a GFP_DMA
> >allocation that eat the last dma page, but we should not kill tasks if
> >we fail to balance in kswapd, we should kill tasks only when no fail
> >path exists (i.e. only during page faults, everything else in the kernel
> >has a fail path and it should never trigger oom).
> >
> >If you move it in kswapd there's no way to prevent oom-killing from a
> >syscall allocation (I guess even right now it would go wrong in this
> >sense, but at least right now it's more fixable). I want to move the oom
> >kill outside the alloc_page paths. The oom killing is all about the page
> >faults not having a fail path, and in turn the oom killing should be
> >moved in the page fault code, not in the allocator. Everything else
> >should keep returning -ENOMEM to the caller.
> >
> >
> 
> Probably a good idea. OTOH, some kernel allocations might really
> need to be performed and have no failure path. For example __GFP_REPEAT.
> 
> I think maybe __GFP_REPEAT allocations at least should be able to
> cause an OOM. Not sure though.

As I said in my answer to Andrea, OOM killing is about allocations not being 
able to succeed (aka VM not being able to free pages).

kswapd is freeing pages, he is the one who knows about progress.

> >So to me moving the oom killer into kswapd looks a regression.
> 
> Also, I think it would do the wrong thing on NUMA machines because
> that has a per-node kswapd.

Right, we need to handle the NUMA case correctly (we need to which does "dont kill if other nodes
have available memory").

But still, it looks the right thing to do to me.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Thomas Gleixner]

On Sat, 2004-11-06 at 02:20 +0100, Andrea Arcangeli wrote:
> On Fri, Nov 05, 2004 at 03:32:50PM -0800, Jesse Barnes wrote:
> > On Friday, November 05, 2004 12:01 pm, Marcelo Tosatti wrote:
> > > In my opinion the correct approach is to trigger the OOM killer
> > > when kswapd is unable to free pages. Once that is done, the number
> > > of tasks inside page reclaim is irrelevant.
> > 
> > That makes sense.
> 
> I don't like it, kswapd may fail balancing because there's a GFP_DMA
> allocation that eat the last dma page, but we should not kill tasks if
> we fail to balance in kswapd, we should kill tasks only when no fail
> path exists (i.e. only during page faults, everything else in the kernel
> has a fail path and it should never trigger oom).
> 
> If you move it in kswapd there's no way to prevent oom-killing from a
> syscall allocation (I guess even right now it would go wrong in this
> sense, but at least right now it's more fixable). I want to move the oom
> kill outside the alloc_page paths. The oom killing is all about the page
> faults not having a fail path, and in turn the oom killing should be
> moved in the page fault code, not in the allocator. Everything else
> should keep returning -ENOMEM to the caller.
> 
> So to me moving the oom killer into kswapd looks a regression.

My point is not where oom-killer is triggered. My point is the decision
criteria of oom-killer, when it is finally invoked, which process to
kill. That's kind of independend of your patch. Your patch corrects the
context in which oom-killer is called. My concern is that the decision
critrion which process should be killed is not sufficient. In my case it
kills sshd instead of a process which forks a bunch of child processes.
Thats just wrong, because it takes away the chance to log into the
machine remotely and fix the problem.

tglx


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Marcelo Tosatti]

On Sat, Nov 06, 2004 at 02:55:50AM +0100, Thomas Gleixner wrote:
> On Sat, 2004-11-06 at 02:20 +0100, Andrea Arcangeli wrote:
> > On Fri, Nov 05, 2004 at 03:32:50PM -0800, Jesse Barnes wrote:
> > > On Friday, November 05, 2004 12:01 pm, Marcelo Tosatti wrote:
> > > > In my opinion the correct approach is to trigger the OOM killer
> > > > when kswapd is unable to free pages. Once that is done, the number
> > > > of tasks inside page reclaim is irrelevant.
> > > 
> > > That makes sense.
> > 
> > I don't like it, kswapd may fail balancing because there's a GFP_DMA
> > allocation that eat the last dma page, but we should not kill tasks if
> > we fail to balance in kswapd, we should kill tasks only when no fail
> > path exists (i.e. only during page faults, everything else in the kernel
> > has a fail path and it should never trigger oom).
> > 
> > If you move it in kswapd there's no way to prevent oom-killing from a
> > syscall allocation (I guess even right now it would go wrong in this
> > sense, but at least right now it's more fixable). I want to move the oom
> > kill outside the alloc_page paths. The oom killing is all about the page
> > faults not having a fail path, and in turn the oom killing should be
> > moved in the page fault code, not in the allocator. Everything else
> > should keep returning -ENOMEM to the caller.
> > 
> > So to me moving the oom killer into kswapd looks a regression.
> 
> My point is not where oom-killer is triggered. My point is the decision
> criteria of oom-killer, when it is finally invoked, which process to
> kill. That's kind of independend of your patch. Your patch corrects the
> context in which oom-killer is called. My concern is that the decision
> critrion which process should be killed is not sufficient. In my case it
> kills sshd instead of a process which forks a bunch of child processes.
> Thats just wrong, because it takes away the chance to log into the
> machine remotely and fix the problem.

Hi Thomas,

Yes your patches are correct and needed independantly of where OOM killer 
is triggered from.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Werner Almesberger]

Entering an old discussion ...

Thomas Gleixner wrote:
> context in which oom-killer is called. My concern is that the decision
> critrion which process should be killed is not sufficient. In my case it
> kills sshd instead of a process which forks a bunch of child processes.

It recently occurred to me that we could have relatively light-weight
voluntary victimization for known trouble-makers. E.g. in a desktop
environment, the cause for trouble seems to be almost always the Web
browser, or something closely related to it.

A process could declare itself as usual suspect. This would then be
recorded as a per-task flag, to be inherited by children. Now, one
could write a launcher like this:

int main(int argc,char **argv)
{
    if (argc < 2) {
	fprintf(stderr,"usage: %s command [arguments...]\n",*argv);
	return 1;
    }
    sys_suspect_me();
    execvp(argv[1],argv+1);
    perror(argv[1]);
    return 1;
}

And then something like

# mv /usr/bin/browser /usr/bin/browser.bin
# echo '#!/bin/sh' >/usr/bin/browser
# echo 'suspect_me /usr/bin/browser.bin "$@"' >>/usr/bin/browser
# chmod 555 /usr/bin/browser

or use an alias if you like your packet manager.

Not sure if this would actually be useful in real life, but it looks
at least like a relatively simple and flexible solution to a part of
the selection problem.

One could even consider getting rid of the suspects a while before
hitting OOM, so that the system doesn't have to slow down before the
inevitable killing.

Not that'm getting many OOMs these days - my VNC setup is quite good
at dying well before anything serious turns up :-(

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Chris Ross]

Werner Almesberger escreveu:
> A process could declare itself as usual suspect. This would then be
> recorded as a per-task flag, to be inherited by children.

I don't think this "I know I'm buggy, please kill me" flag is the right 
approach even if it can be made to work. The operating system has an 
overview of all the memory and can see when a particular process is 
basically making the machine unusable. It's quite likely that the 
process causing the trouble doesn't know (or hasn't admitted) that it's 
buggy and hasn't volunteered for early termination. As this means the 
kernel must be able to deal with a problematic process completely 
irrespective of whether it has set "kill me" flag or not the flag 
doesn't really buy you anything.

It is also specific to runaway processes that are clearly at fault. 
There is the related case where no particular process is faulty as such 
but the system as a whole can't cope with the demands being made.

On a related note, I would prefer to see victim processes who are not 
determined to be the cause of the trouble swapped out (i.e. *all* their 
pages pushed out to swap) and suspended (not allowed to run) as a first 
resort. The example I have in mind is on my machine when the daily cron 
run over commits causing standard daemons such as ntpd to be killed to 
make room. It would be preferable if the daemon was swapped out and just 
didn't run for minutes, or even hours if need be, but was allowed to run 
again once the system had settled down.

Of course, from recent discussion the system should not actually be 
killing off these daemons at all but that does seem to be resolved now. 
There are circumstances when there simply isn't enough RAM and swapping 
something out is preferable to killing it off. Of course, if there isn't 
sufficient swap space killing it should be the second resort. The last 
resort being panic.

So, the problem breaks down into three parts:

	  i) When should the oom killer be invoked.
	 ii) How do we pick a victim process
	iii) How can we deal with the process in the most useful manner

Regards,
Chris R.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Werner Almesberger]

Chris Ross wrote:
> The operating system has an 
> overview of all the memory and can see when a particular process is 
> basically making the machine unusable.

The underlying hypothesis for suggesting explicitly flagging
candidates for killing is of course that it doesn't see who
exactly is misbehaving :-) Since this issue has been around for
a nummber of years, I think it's fair to assume that the OOM
killers indeed have a problem in that area.

> It's quite likely that the 
> process causing the trouble doesn't know (or hasn't admitted) that it's 
> buggy and hasn't volunteered for early termination.

I guess that depends a lot on your scenario. If your system is
the typical undergrad mainframe where an army of students is
hard at work trying to fork-bomb it out of existence, you're
absolutely right.

However, on a system where new programs are rarely added to the
mix, the distinction should be easier. You can still get
unexpected problems, e.g. vi trying to load a huge file, but
you should be in a much better position to profile your system
behaviour.

It could of course be that this scenario is overly specific.

> As this means the 
> kernel must be able to deal with a problematic process completely 
> irrespective of whether it has set "kill me" flag or not the flag 
> doesn't really buy you anything.

I'd view it as an additional hint that killing that process is
likely to help, a) because it may be the culprit, or b) because
it is likely to hold lots of memory, and its death will not be
mourned.

I'm not suggesting to use this as a replacement for an adaptive
OOM killer. The OOM killer could first make its pick among the
suspects, and only if it runs out of them (or maybe if it finds
overwhelming evidence that it's something else), then it would
go after non-suspects.

> There is the related case where no particular process is faulty as such 
> but the system as a whole can't cope with the demands being made.

Yes, that's yet another scenario. Even then, having a list of
things we can kill to give us some room would be useful.

> The example I have in mind is on my machine when the daily cron 
> run over commits causing standard daemons such as ntpd to be killed to 
> make room. It would be preferable if the daemon was swapped out and just 
> didn't run for minutes, or even hours if need be, but was allowed to run 
> again once the system had settled down.

Ah, now I understand why you'd want to swap. Interesting. Now,
depending on the time if day, you have typically "interactive"
processes, like your idle desktop, turn into "non-interactive"
ones, which can then be subjected to swapping. Nice example
against static classification :-)

> So, the problem breaks down into three parts:
> 
> 	  i) When should the oom killer be invoked.
> 	 ii) How do we pick a victim process
> 	iii) How can we deal with the process in the most useful manner

iii) may also affect i). If you're going to swap, you don't want
to wait until you're fighting for the last available page in the
system.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Chris Ross]

Werner Almesberger escreveu:
> Chris Ross wrote:
> The underlying hypothesis for suggesting explicitly flagging
> candidates for killing is of course that it doesn't see who
> exactly is misbehaving :-) Since this issue has been around for
> a nummber of years, I think it's fair to assume that the OOM
> killers indeed have a problem in that area.

That's my point ii) below, which is what Thomas's patch is trying to 
address. I doubt you'd find much disagreement that this area still needs 
work :)

>>The example I have in mind is on my machine when the daily cron 
>>run over commits causing standard daemons such as ntpd to be killed to 
>>make room. It would be preferable if the daemon was swapped out and just 
>>didn't run for minutes, or even hours if need be, but was allowed to run 
>>again once the system had settled down.
> 
> Ah, now I understand why you'd want to swap. Interesting. Now,
> depending on the time if day, you have typically "interactive"
> processes, like your idle desktop, turn into "non-interactive"
> ones, which can then be subjected to swapping. Nice example
> against static classification :-)

A better example than the ntpd daemon (which mightn't take kindly to 
finding minutes just passed in a blink of its eye) is Thomas's example 
with the sshd. If the daemon was swapped out you wouldn't be able to log 
into the box while it was thrashing, but in practice you can't really 
anyway. At least once the system had recovered sufficiently you could 
get back in, under the present system you can never log in again.

>>So, the problem breaks down into three parts:
>>
>>	  i) When should the oom killer be invoked.
>>	 ii) How do we pick a victim process
>>	iii) How can we deal with the process in the most useful manner
> 
> iii) may also affect i). If you're going to swap, you don't want
> to wait until you're fighting for the last available page in the
> system.

Well yes, in typical fashion everything depends on everything else. That 
in a nutshell is also my argument against the kill-me flag.

Regards,
Chris R.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Werner Almesberger]

Chris Ross wrote:
> with the sshd. If the daemon was swapped out you wouldn't be able to log 
> into the box while it was thrashing, but in practice you can't really 
> anyway.

Nor would you want to, in the scenario you're describing, because
the system is doing housekeeping while you're away/asleep. I
agree that this makes sense.

The tricky bit is now to identify such part-time interactive tasks,
i.e. the ones who won't receive a trigger for a while. To make
things worse, there are those who may be happily doing something,
like spinning some animated GIF, which would be perfectly fine
being put to a long sleep. That in turn may make the X server idle,
etc.

Again, if you have such a clearly defined scenario, perhaps the
cron jobs should just loudly announce that housekeeping is now
starting and that this changes some of the rules. Or perhaps,
there could be a SIGSWAP to swap out a process (maybe SIGSUSP it
first so that it doesn't come back on its own).

> Well yes, in typical fashion everything depends on everything else. That 
> in a nutshell is also my argument against the kill-me flag.

I think it may be more subtle: everybody seems to have a set of
scenarios where the best solution is quite obvious and could
be easily implemented. Also, every once in a while, you find
that system loads which clearly demand a specific action in
scenario A need something very different in scenario B.

E.g. if you go by load spike, you'll be able to contain some
of the less inspired experiments on that undergrad mainframe,
but you may end up killing the cron jobs in your housekeeping
scenario. (And in this case, swapping wouldn't even help.) Or,
if you never kill anything big with a long run time, you'll
protect that simulation of an universe that's just on the
verge of developing intelligent life, but you may completely
miss the Web browser that's been rotating banner ads for weeks.
(Here, swapping might help.)

So I think that you also need to know what the processes are,
and not only what they're doing. This should greatly improve
predictions of what they will do in the future, and why
they're doing it in the first place.

It's ultimately policy decisions, and that's where I see a place
for light-weight markup mechanisms like a "kill me first" bit.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Chris Ross]

Werner Almesberger escreveu:
> The tricky bit is now to identify such part-time interactive tasks,
> i.e. the ones who won't receive a trigger for a while. To make
> things worse, there are those who may be happily doing something,
> like spinning some animated GIF, which would be perfectly fine
> being put to a long sleep. That in turn may make the X server idle,
> etc.

I don't think you need to be that subtle about it, though I agree 
perfection would be nice :) The present behaviour is just to kill 
something. All I'm advocating is just swapping something out if possible 
instead. Yes by definition we probably have picked something you would 
have preferred to leave running, but the machine simply cope with 
everything being asked of it at the moment and that something got the 
short straw. At least swapped out we will get round to running it when 
we can.

> Again, if you have such a clearly defined scenario, perhaps the
> cron jobs should just loudly announce that housekeeping is now
> starting and that this changes some of the rules. Or perhaps,
> there could be a SIGSWAP to swap out a process (maybe SIGSUSP it
> first so that it doesn't come back on its own).

Sounds like a job for priorities and sensible use of batch scheduling.

I still feel that special casing things is basically wrong. We could 
work around the specific example that the cron.daily on my test machines 
tends to cause things to be oom_killed, but it's better to fix the 
problem. What about when I try to build umlsim again -- my standard test 
case for triggering the oom killer ;)

Let's not forget that oom killing (when it works) is a last resort, 
something we do only if we have to to avoid a panic. Too often at 
present the machine just doesn't know what to do, runs around confused 
and makes things worse by shooting its own leg off. Which is pretty much 
a real-world definition of panicking*. Lets at least try to avoid 
causing permanent damage, such as killing off sshd.

[ * I just looked it up: "of, relating to, or resembling the mental or 
emotional state believed induced by the god Pan". Cool ]

Regards,
Chris R.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Werner Almesberger]

Chris Ross wrote:
> All I'm advocating is just swapping something out if possible 
> instead.

Yes, but this only works if a) your system can make progress
towards lowering its memory needs without the process(es) you've
picked for swapping, and b) these processes don't happen to be
something that cannot tolerate long suspension, and c) the total
memory needs are such that they can be better satisfied after
these processes have been swapped out.

Examples where this isn't the case: a) if you swap out your
hoursekeeping cron job, the system will just sit idle, then you
swap it in again after a few minutes, and the agony repeats.
b) if you swap out my X server while I'm sitting at the machine,
all you've done was to force me to press the big red switch
manually. c) if there's a process with excessive memory demands
that can't be met anyway, it's better to end its misery quickly,
instead of spending a day thrashing.

So again, your automatic OOM kill^H^H^H^Hcounsellor doesn't only
have to follow a fixed policy, but it also has to sense what kind
of situation we're in.

A SIGSWAP would help with a) and b). In case a), the cron jobs
would signal anything that's not them. In case b), by definition,
I'd not be working when this happens. This can be assisted by
user detection heuristics as used in some batch distribution
systems. (Now we have a fairly complex user space already, with
lots of policy.) The usual "runaway process" heuristics can
probably take care of c).

> Too often at present the machine just doesn't know what to do,

See, that's exactly what I mean :-) So, why not just tell it ?
"Hey, things are going to get a little rough for a while. Why
don't you take a nap on that comfty swap disk while I clean up
the house ?"

> [ * I just looked it up: "of, relating to, or resembling the mental or 
> emotional state believed induced by the god Pan". Cool ]

Hmm, you're suggesting we follow Morpheus instead of Pan then ?
And I always thought the OOM killer was more like Eris' work :-)

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Thomas Gleixner]

On Thu, 2004-11-18 at 07:01 -0300, Werner Almesberger wrote:
> > Too often at present the machine just doesn't know what to do,
> 
> See, that's exactly what I mean :-) So, why not just tell it ?
> "Hey, things are going to get a little rough for a while. Why
> don't you take a nap on that comfty swap disk while I clean up
> the house ?"
> 
> > [ * I just looked it up: "of, relating to, or resembling the mental or 
> > emotional state believed induced by the god Pan". Cool ]
> 
> Hmm, you're suggesting we follow Morpheus instead of Pan then ?
> And I always thought the OOM killer was more like Eris' work :-)

Hmm, what about embedded boxes without swap ? There you have only one
choice. Kill anything appropriate. 

I tested Marcellos, Andreas and Andrews changes and they all change the
trigger of the oom-killer to prevent the spurious unneccecary kills. But
in the case where an oom-kill is neccecary I still need my modifications
to the whom to kill decision and to the oom-killer itself.

1. Not to kill the innocent and maybe important processes like sshd

This one is solved by taking the childs into account.

2. To prevent overkill. 

This still happens with all the modifications I have tested. 
The szenario is that two processes request memory in an OOM situation.
The first caller to oom-kill is making room and then the second one is
also going to kill something. 

The oom-killer must be protected against reentrancy and it must check
for the free memory level before finally doing the kill.
See my previous patch in this thread.

BTW, I found out that hackbench is a quite good replacement for my real
application in triggering these corner cases. Just increase the number
of processes until you reach the limit of the machine.

tglx


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Chris Friesen]

Thomas Gleixner wrote:

> Hmm, what about embedded boxes without swap ? There you have only one
> choice. Kill anything appropriate. 

I worked on a project that took the opposite approach from the "I'm a suspect" 
flag mentioned earlier.  Processes could request immunity from the OOM killer as 
long as they were under a specified memory usage.  Critical apps were thus 
protected as long as they were sane, while noncritical stuff could be killed at 
will.

Chris
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Marcelo Tosatti]

On Sat, Nov 06, 2004 at 02:20:18AM +0100, Andrea Arcangeli wrote:
> On Fri, Nov 05, 2004 at 03:32:50PM -0800, Jesse Barnes wrote:
> > On Friday, November 05, 2004 12:01 pm, Marcelo Tosatti wrote:
> > > In my opinion the correct approach is to trigger the OOM killer
> > > when kswapd is unable to free pages. Once that is done, the number
> > > of tasks inside page reclaim is irrelevant.
> > 
> > That makes sense.

Hi Andrea,

> I don't like it, kswapd may fail balancing because there's a GFP_DMA
> allocation that eat the last dma page, but we should not kill tasks if
> we fail to balance in kswapd, we should kill tasks only when no fail
> path exists (i.e. only during page faults, everything else in the kernel
> has a fail path and it should never trigger oom).

The OOM killer is only going to get triggered if kswapd is not able 
to make _any_ progress in all zones.  So it wont "fail balancing because there's 
a GFP_DMA allocation that eat the last dma page".  

As long as frees _one_ page during all passes from DEF_PRIORITY till priority=0, 
it wont kill any task. See?

I dont get your point.

> If you move it in kswapd there's no way to prevent oom-killing from a
> syscall allocation (I guess even right now it would go wrong in this
> sense, but at least right now it's more fixable).

I dont understand what you mean. "prevent oom-killing from a syscall allocation" ?

>  I want to move the oom
> kill outside the alloc_page paths. The oom killing is all about the page
> faults not having a fail path, and in turn the oom killing should be
> moved in the page fault code, not in the allocator. Everything else
> should keep returning -ENOMEM to the caller.

Isnt OOM killing all about the reclaiming efforts not being able to make progress? 

> So to me moving the oom killer into kswapd looks a regression. 

To me having tasks trigger the OOM kill is fundamentally broken 
because it doesnt take into account kswapd page freeing 
efforts which are in-progress at the very moment.

That makes senses a lot of sense to me - would love to be proved
wrong.

See, its completly screwed right now. The code inside out_of_memory()
which only triggers OOM if it has happened several times during the 
past few seconds is horrible and shows how bad it is. 
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Andrea Arcangeli]

Hi Marcelo,

On Sat, Nov 06, 2004 at 08:05:16AM -0200, Marcelo Tosatti wrote:
> The OOM killer is only going to get triggered if kswapd is not able 
> to make _any_ progress in all zones.  So it wont "fail balancing because there's 
> a GFP_DMA allocation that eat the last dma page".  
> 
> As long as frees _one_ page during all passes from DEF_PRIORITY till priority=0, 
> it wont kill any task. See?

It's still wrong on numa. the machine isn't oom despite kswapd couldn't
free any page (the local node will fallback in the other nodes instead)

> > If you move it in kswapd there's no way to prevent oom-killing from a
> > syscall allocation (I guess even right now it would go wrong in this
> > sense, but at least right now it's more fixable).
> 
> I dont understand what you mean. "prevent oom-killing from a syscall allocation" ?

yes. oom killing should be avoided as far as we can avoid it. Ideally we
should never invoke the oom killer and we should always return -ENOMEM
to applications. If a syscall runs oom then we can return -ENOMEM and
handle the failure gracefully instead of getting a sigkill.

With 2.4 -ENOMEM is returned and the machine doesn't deadlock when the
zone normal is full and that works fine.

> Isnt OOM killing all about the reclaiming efforts not being able to make progress? 

it's invoked when we're not able to make progress and no fail path
exists.

> To me having tasks trigger the OOM kill is fundamentally broken 
> because it doesnt take into account kswapd page freeing 
> efforts which are in-progress at the very moment.

kswapd page freeing efforts are not very useful. kswapd is an helper,
it's not the thing that can or should guarantee allocations to succeed.

The rule is that if you want to allocate 1 page, you've to free the page
yourself. Then if kswapd frees a page too, that's welcome. But keep also
in mind kswapd may be running in another cpu, and it will put the pages
back into the per-cpu queue of the other cpu. So you should really free
a page yourself to be guaranteed to find that page later on.

kswapd is more for keeping the balance between low and high so that we
never block freeing memory, and to keep the disk running.

> See, its completly screwed right now. The code inside out_of_memory()
> which only triggers OOM if it has happened several times during the 
> past few seconds is horrible and shows how bad it is. 

that's very bad indeed. But anything happening inside out_of_memory has
nothing to do with what we discussed above like Thomas Gleixner pointed
out yesterday.

these are two different things:

1) choose when we need to invoke out_of_memory
2) choose what to do inside out_of_memory

I definitely agree about the 5 sec waiting being an hack, to me even
blk_congest_wait looks an hack.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Arjan van de Ven]

> yes. oom killing should be avoided as far as we can avoid it. Ideally we
> should never invoke the oom killer and we should always return -ENOMEM
> to applications. If a syscall runs oom then we can return -ENOMEM and
> handle the failure gracefully instead of getting a sigkill.
> 
> With 2.4 -ENOMEM is returned and the machine doesn't deadlock when the
> zone normal is full and that works fine.

the harder case is where you do an mmap and then in the fault path find out that there's no memory to allocate the PMD ...
killing the task that has that failing isn't per se the right answer.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Marcelo Tosatti]

On Sat, Nov 06, 2004 at 04:44:15PM +0100, Andrea Arcangeli wrote:
> Hi Marcelo,

Hi again!

> On Sat, Nov 06, 2004 at 08:05:16AM -0200, Marcelo Tosatti wrote:
> > The OOM killer is only going to get triggered if kswapd is not able 
> > to make _any_ progress in all zones.  So it wont "fail balancing because there's 
> > a GFP_DMA allocation that eat the last dma page".  
> > 
> > As long as frees _one_ page during all passes from DEF_PRIORITY till priority=0, 
> > it wont kill any task. See?
> 
> It's still wrong on numa. the machine isn't oom despite kswapd couldn't
> free any page (the local node will fallback in the other nodes instead)

Sure NUMA can and has to be special cased, as I answered Nick. 

"dont kill if we can allocate from other nodes", should be pretty simple.

> > > If you move it in kswapd there's no way to prevent oom-killing from a
> > > syscall allocation (I guess even right now it would go wrong in this
> > > sense, but at least right now it's more fixable).
> > 
> > I dont understand what you mean. "prevent oom-killing from a syscall allocation" ?
> 
> yes. oom killing should be avoided as far as we can avoid it. Ideally we
> should never invoke the oom killer and we should always return -ENOMEM
> to applications. If a syscall runs oom then we can return -ENOMEM and
> handle the failure gracefully instead of getting a sigkill.
> 
> With 2.4 -ENOMEM is returned and the machine doesn't deadlock when the
> zone normal is full and that works fine.

I agree with you here. But then there are the cases which you can't return ENOMEM - 
the fault paths you have mentioned, and the PMD/PTE allocation mentioned by Arjan
on mmap, and probably others.

We should be returning -ENOMEM to syscalls right now in v2.6, but thats 
not the problem here. The problem are the page faults.

If v2.6 is failing to return -ENOMEM to syscalls then its indeed screwed, 
but its not the same problem.

Have you done any tests on this respect?

> > Isnt OOM killing all about the reclaiming efforts not being able to make progress? 
> 
> it's invoked when we're not able to make progress and no fail path
> exists.

The system will, in the vast majority of cases, be OOM due to page faults
which can't be handled (anonymous memory mappings created by brk/sbrk) anyway.

So "not being able to make progress freeing pages" seems to be reliable
information on whether to trigger OOM. Note that reaching priority
0 means we've tried VERY VERY hard already.

> > To me having tasks trigger the OOM kill is fundamentally broken 
> > because it doesnt take into account kswapd page freeing 
> > efforts which are in-progress at the very moment.
> 
> kswapd page freeing efforts are not very useful. kswapd is an helper,
> it's not the thing that can or should guarantee allocations to succeed.

Oh wait, kswapd job is to guarantee that allocations succeed. We used to 
wait on kswapd before on v2.3 VM development - then we switched to 
task-goes-to-memory-reclaim for _performance_ reasons (parallelism).

My point here is, kswapd is the entity responsible for freeing pages. 

The action of triggering OOM killer from inside a task context (whether 
its from the alloc_pages path or the fault path is irrelevant here) 
is WRONG because at the same time, kswapd, who is the main entity freeing 
pages, is also running the memory reclaim code - it might just have freed 
a bunch of pages but we have no way of knowing that from normal task context. 

> The rule is that if you want to allocate 1 page, you've to free the page
> yourself. Then if kswapd frees a page too, that's welcome. But keep also
> in mind kswapd may be running in another cpu, and it will put the pages
> back into the per-cpu queue of the other cpu. 

Exactly another reason for _NOT_ triggering the OOM killer from task context 
- pages which have been freed might be in the per-CPU queue (but a task
running on another CPU can't see them).

We should be flushing the per-cpu queues quite often under these circumstances. 

> So you should really free
> a page yourself to be guaranteed to find that page later on.
> 
> kswapd is more for keeping the balance between low and high so that we
> never block freeing memory, and to keep the disk running.
> 
> > See, its completly screwed right now. The code inside out_of_memory()
> > which only triggers OOM if it has happened several times during the 
> > past few seconds is horrible and shows how bad it is. 
> 
> that's very bad indeed. But anything happening inside out_of_memory has
> nothing to do with what we discussed above like Thomas Gleixner pointed
> out yesterday.
> 
> these are two different things:
> 
> 1) choose when we need to invoke out_of_memory
> 2) choose what to do inside out_of_memory

OK - agreed.

> I definitely agree about the 5 sec waiting being an hack, to me even
> blk_congest_wait looks an hack.

Why's that? blk_congestion_wait looks clean and correct to me - if the queue 
is full dont bother queueing more pages at this device.

OK - so you seem to be agreeing with me that triggering OOM killer
from kswapd context is the correct thing to do now?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Andrea Arcangeli]

On Sat, Nov 06, 2004 at 03:09:25PM -0200, Marcelo Tosatti wrote:
> Sure NUMA can and has to be special cased, as I answered Nick. 
> 
> "dont kill if we can allocate from other nodes", should be pretty simple.

then how do you call the oom killer if some highmem page is still free
but there's a GFP_KERNEL|__GFP_NOFAIL allocation failing and looping
forever?

> If v2.6 is failing to return -ENOMEM to syscalls then its indeed screwed, 
> but its not the same problem.
> 
> Have you done any tests on this respect?

the only test I have are based on 2.6.5 and the box simply deadlock
there, the oom killer is forbidden there if nr_swap_pages > 0. Not
anymore in 2.6.9rc, however with 2.6.9 and more recent the oom killer is
invoked too early.

> So "not being able to make progress freeing pages" seems to be reliable
> information on whether to trigger OOM. Note that reaching priority
> 0 means we've tried VERY VERY hard already.

yes, "not being able to make progress freeing pages" has always been the
only reliable information in linux. The early 2.6 and some older 2.4
deadlocked in corner cases (like mlock for example) at trying to guess
the oom time by looking at a few stat numbers (nr_swap_pages for
example).

Though when we reach prio 0 clearly we didn't try hard enough if I'm
getting spurious oom. It's also possible that the pages are being freed
in other per-cpu queues and we lose visibility on them, so we do hard
work and still we cannot allocate anything. Unfortunately I've never
been able to reproduce the early oom kill here, so I could never figure
out what's going on. and yes, I know there are patches floating around
claiming they fixed it, but I'd like to hear positive feedback on those.

> > kswapd page freeing efforts are not very useful. kswapd is an helper,
> > it's not the thing that can or should guarantee allocations to succeed.
> 
> Oh wait, kswapd job is to guarantee that allocations succeed. We used to 

kswapd is all but a guarantee. kswapd is a pure helper.

> wait on kswapd before on v2.3 VM development - then we switched to 
> task-goes-to-memory-reclaim for _performance_ reasons (parallelism).

it wasn't parallelism.  The only way you could make it safe is that you
create a message passing mechanism where you post a request to kswapd
and kswapd wakes you up back. But that'd inefficient compared to current
model where kswapd is an helper.

Infact kswapd right now only hurts during heavy paging since it will
prevent the freed pages to go into the right per-cpu queue. kswapd only
hurts during paging, we should stop it as far as somebody is inside the
critical section for a certain numa node.

> My point here is, kswapd is the entity responsible for freeing pages. 

it can't even know which is the per-cpu queue where it has to put the
pages back.

> The action of triggering OOM killer from inside a task context (whether 
> its from the alloc_pages path or the fault path is irrelevant here) 
> is WRONG because at the same time, kswapd, who is the main entity freeing 
> pages, is also running the memory reclaim code - it might just have freed 
> a bunch of pages but we have no way of knowing that from normal task context. 

we definitely have a way of knowing, the fact the current code is buggy
doesn't mean we don't have a way of knowing, 2.4 VM perfectly knows when
kswapd did the right thing and helped. Though I agree kswapd generally
hurts during paging and we'd better stop it to reduce the synchronous
amount of work.

the allocator must check if the levels are above pages.low before
killing, if it doesn't do that it's broken, moving the oom killer in
kswapd cannot fix this problem, because obviously then it'll be the task
context that will have freed the additional pages instead of kswapd.

The rule is to do:

	paging
	check the levels and kill

If you just do paging and oom kill if paging has failed there's no way
it can work. 2.6 is broken here, and that could be the reason of the oom
kills too.

There will be always a race condition even with the above, since the
check for the levels and oom kill isn't an atomic operation and we don't
block all other cpus in that path, but it's an insignificant window we
don't have to worry about (only theoretical).

But if you check the levels; paging; kill, like current 2.6, there is an
huge window while we wait for I/O. After we finished the I/O the whole
VM status may have changed and we may be full of free pages in the
per-cpu queue and in the buddy as well. so we've to recheck the levels
before killing anything. This is again why doing oom_kill inside the
try_to_free_pages (or in kswapd anyways) is flawed.

> > The rule is that if you want to allocate 1 page, you've to free the page
> > yourself. Then if kswapd frees a page too, that's welcome. But keep also
> > in mind kswapd may be running in another cpu, and it will put the pages
> > back into the per-cpu queue of the other cpu. 
> 
> Exactly another reason for _NOT_ triggering the OOM killer from task context 
> - pages which have been freed might be in the per-CPU queue (but a task
> running on another CPU can't see them).
> 
> We should be flushing the per-cpu queues quite often under these circumstances. 

we should never flush per-cpu pages, that'd hurt performance, per-cpu
pages are lost memory. this is also why we must give up freeing memory
only if everything else is not available, in 2.4 I had to stop after 50
tries or so. We must keep going until all per-cpu queues are full,
because if kswapd is in our way every other cpu may get the ram before
us. This is why stopping kswapd would be beneficial while we're working
on it, it'd probabilistically reduce the amount of synchronous work.

> Why's that? blk_congestion_wait looks clean and correct to me - if the queue 
> is full dont bother queueing more pages at this device.

blk_contestion_wait is waiting on random I/O, it doesn't mean it's
waiting on any substantial VM related paging (an O_DIRECT I/O would fool
blk_congestion_wait), and if there's no I/O it just wakeup after a fixed
random number of seconds.

the VM should only throttle on locked pages or locked bh it can see,
never on random I/O happening at the blkdev layer just because somebody
is rolling some directio. Throttling on random I/O will lead to oom
failures too and that's another bug in the 2.6 VM (and if it's not a
bug, and we throttle elsewhere too, then it's simply useless and it
should be replaced with a yield, if there's no I/O waiting there is a
nosense, especially given that even if the oom killer triggers there
won't be any additional ram to free, since the oom killer will generate
free memory, not memory to free).

> OK - so you seem to be agreeing with me that triggering OOM killer
> from kswapd context is the correct thing to do now?

I disagree about that sorry. not even try_to_free_pages should ever call
the oom killer (unless you want to move the watermark checks from
page_alloc.c to vmscan.c that would not be clean at all). Taking the
decision on when to oom kill inside vmscan.c (like current 2.6 does)
looks wrong to me.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Marcelo Tosatti]

On Sun, Nov 07, 2004 at 01:48:09AM +0100, Andrea Arcangeli wrote:
> On Sat, Nov 06, 2004 at 03:09:25PM -0200, Marcelo Tosatti wrote:
> > Sure NUMA can and has to be special cased, as I answered Nick. 
> > 
> > "dont kill if we can allocate from other nodes", should be pretty simple.
> 
> then how do you call the oom killer if some highmem page is still free
> but there's a GFP_KERNEL|__GFP_NOFAIL allocation failing and looping
> forever?

We should probably kill a task if a GFP_KERNEL allocation 
is failing and looping forever. We need some mechanism
to indicate to kswapd that there is a allocator 
failing repeatedly (an indication that we are really getting into
an OOM condition).

With that information kswapd can decide whether to kill or not. 

> > but its not the same problem.
> > 
> > Have you done any tests on this respect?
> 
> the only test I have are based on 2.6.5 and the box simply deadlock
> there, the oom killer is forbidden there if nr_swap_pages > 0. Not
> anymore in 2.6.9rc, however with 2.6.9 and more recent the oom killer is
> invoked too early.

Right. 

> > So "not being able to make progress freeing pages" seems to be reliable
> > information on whether to trigger OOM. Note that reaching priority
> > 0 means we've tried VERY VERY hard already.
> 
> yes, "not being able to make progress freeing pages" has always been the
> only reliable information in linux. The early 2.6 and some older 2.4
> deadlocked in corner cases (like mlock for example) at trying to guess
> the oom time by looking at a few stat numbers (nr_swap_pages for
> example).

Right, and William removed the "nr_swap_pages" check, which fixed 
those - but introduced the spurious OOM kills.

> Though when we reach prio 0 clearly we didn't try hard enough if I'm
> getting spurious oom. It's also possible that the pages are being freed
> in other per-cpu queues and we lose visibility on them, so we do hard
> work and still we cannot allocate anything. Unfortunately I've never
> been able to reproduce the early oom kill here, so I could never figure
> out what's going on. and yes, I know there are patches floating around
> claiming they fixed it, but I'd like to hear positive feedback on those.

Just run a program which allocates a lot of anonymous memory on a 100M setup
and you will see it. 

The OOM killer will be triggered even if there is anonymous memory to be 
swapped out and swap space available. Which is plain wrong.

> > > kswapd page freeing efforts are not very useful. kswapd is an helper,
> > > it's not the thing that can or should guarantee allocations to succeed.
> > 
> > Oh wait, kswapd job is to guarantee that allocations succeed. We used to 
> 
> kswapd is all but a guarantee. kswapd is a pure helper.
> 
> > wait on kswapd before on v2.3 VM development - then we switched to 
> > task-goes-to-memory-reclaim for _performance_ reasons (parallelism).
> 
> it wasn't parallelism.  The only way you could make it safe is that you
> create a message passing mechanism where you post a request to kswapd
> and kswapd wakes you up back. But that'd inefficient compared to current
> model where kswapd is an helper.

Yes, it was inefficient, but it used to work like that. There was a
kswapd waitqueue in which tasks used to sleep. By parallelism I meant 
parallelism freeing pages -> efficiency.  

OK - its not like that anymore - but kswapd is not simply a "helper" - 
its more than that, its responsible for keeping enough pages free.

> Infact kswapd right now only hurts during heavy paging since it will
> prevent the freed pages to go into the right per-cpu queue. kswapd only
> hurts during paging, we should stop it as far as somebody is inside the
> critical section for a certain numa node.

Indeed! But then if you stop kswapd under heavy load, you're pretty much 
guaranteeing tasks will have to free pages themselves - synchronously
, which is not a good thing.

Its not trivial to know when to stop/when to start kswapd. 

Another related problem is that an unlimited number of tasks can go
into reclaim - it should be limited to a sane number.

> > My point here is, kswapd is the entity responsible for freeing pages. 
> 
> it can't even know which is the per-cpu queue where it has to put the
> pages back.

No need - just put send'em back to the buddy allocator - which happens
anyway when the per-cpu queue reaches its high limit. 

> > The action of triggering OOM killer from inside a task context (whether 
> > its from the alloc_pages path or the fault path is irrelevant here) 
> > is WRONG because at the same time, kswapd, who is the main entity freeing 
> > pages, is also running the memory reclaim code - it might just have freed 
> > a bunch of pages but we have no way of knowing that from normal task context. 
> 
> we definitely have a way of knowing, the fact the current code is buggy
> doesn't mean we don't have a way of knowing, 2.4 VM perfectly knows when
> kswapd did the right thing and helped. Though I agree kswapd generally
> hurts during paging and we'd better stop it to reduce the synchronous
> amount of work.
> 
> the allocator must check if the levels are above pages.low before
> killing, if it doesn't do that it's broken, moving the oom killer in
> kswapd cannot fix this problem, because obviously then it'll be the task
> context that will have freed the additional pages instead of kswapd.

OK - the pages.low check before killing makes total sense.

We dont do that right now - and we should - whatever callee 
invokes the OOM killer.

> The rule is to do:
> 
> 	paging
> 	check the levels and kill
> 
> If you just do paging and oom kill if paging has failed there's no way
> it can work. 2.6 is broken here, and that could be the reason of the oom
> kills too.
> 
> There will be always a race condition even with the above, since the
> check for the levels and oom kill isn't an atomic operation and we don't
> block all other cpus in that path, but it's an insignificant window we
> don't have to worry about (only theoretical).
> 
> But if you check the levels; paging; kill, like current 2.6, there is an
> huge window while we wait for I/O. After we finished the I/O the whole
> VM status may have changed and we may be full of free pages in the
> per-cpu queue and in the buddy as well. so we've to recheck the levels
> before killing anything. This is again why doing oom_kill inside the
> try_to_free_pages (or in kswapd anyways) is flawed.

OK, it could be a task who now freed pages instead of kswapd - but I still 
prefer having the OOM killer be called from a centralized place, which is 
also the main responsible task for freeing pages, than to call it from each 
task try_to_free_pages() path. 

Anyway, where do you suggest oom_kill to be called from if you think 
both try_to_free_pages/kswapd are not the right callees? 

Failure of handle_mm_fault ? 

> > > The rule is that if you want to allocate 1 page, you've to free the page
> > > yourself. Then if kswapd frees a page too, that's welcome. But keep also
> > > in mind kswapd may be running in another cpu, and it will put the pages
> > > back into the per-cpu queue of the other cpu. 
> > 
> > Exactly another reason for _NOT_ triggering the OOM killer from task context 
> > - pages which have been freed might be in the per-CPU queue (but a task
> > running on another CPU can't see them).
> > 
> > We should be flushing the per-cpu queues quite often under these circumstances. 
> 
> we should never flush per-cpu pages, that'd hurt performance, per-cpu
> pages are lost memory. this is also why we must give up freeing memory
> only if everything else is not available, in 2.4 I had to stop after 50
> tries or so. We must keep going until all per-cpu queues are full,
> because if kswapd is in our way every other cpu may get the ram before
> us. This is why stopping kswapd would be beneficial while we're working
> on it, it'd probabilistically reduce the amount of synchronous work.

OK - flushing the per-cpu queues would hurt performance. 

I meant, if we're kswapd, send'em back to the buddy allocator 
so that tasks on other CPUs can see the just-freed-memory.  

But thats indeed bad from a performance point of view.

> > Why's that? blk_congestion_wait looks clean and correct to me - if the queue 
> > is full dont bother queueing more pages at this device.
> 
> blk_contestion_wait is waiting on random I/O, it doesn't mean it's
> waiting on any substantial VM related paging (an O_DIRECT I/O would fool
> blk_congestion_wait), and if there's no I/O it just wakeup after a fixed
> random number of seconds.
> 
> the VM should only throttle on locked pages or locked bh it can see,
> never on random I/O happening at the blkdev layer just because somebody
> is rolling some directio. Throttling on random I/O will lead to oom
> failures too and that's another bug in the 2.6 VM (and if it's not a
> bug, and we throttle elsewhere too, then it's simply useless and it
> should be replaced with a yield, if there's no I/O waiting there is a
> nosense, especially given that even if the oom killer triggers there
> won't be any additional ram to free, since the oom killer will generate
> free memory, not memory to free).
> 
> > OK - so you seem to be agreeing with me that triggering OOM killer
> > from kswapd context is the correct thing to do now?
> 
> I disagree about that sorry. not even try_to_free_pages should ever call
> the oom killer (unless you want to move the watermark checks from
> page_alloc.c to vmscan.c that would not be clean at all). Taking the
> decision on when to oom kill inside vmscan.c (like current 2.6 does)
> looks wrong to me.

OK, so, please answer my question above on "where do you think is 
the correct callee".

IMHO OOM killing from kswapd, is way better than the current approach 
- and I will try to enhance my patch handling the NUMA and GFP_KERNEL
allocation cases (with a notification mechanism used by allocators 
to inform successive failure to kswapd).

Either way - this discussion is being productive.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer ...

[Andries Brouwer]

On Fri, Nov 05, 2004 at 06:01:18PM -0200, Marcelo Tosatti wrote:

> My wife is almost killing me, its Friday night and I've been telling her
> "just another minute" for hours. Have to run.

:-)

> As you know the OOM is very problematic in 2.6 right now - so I went
> to investigate it.

I have always been surprised that so few people investigated
doing things right, that is, entirely without OOM killer.
Apparently developers do not think about using Linux for serious work
where it can be a disaster, possibly even a life-threatening disaster,
when any process can be killed at any time.

Ten years ago it was a bad waste of resources to have swapspace
lying around that would be used essentially 0% of the time.
But with todays disk sizes it is entirely feasible to have
a few hundred MB of "unused" swap space. A small price to
pay for the guarantee that no process will be OOM killed.

A month ago I showed a patch that made overcommit mode 2
work for me. Google finds it in http://lwn.net/Articles/104959/

So far, nobody commented.

This is not in a state such that I would like to submit it,
but I think it would be good to focus some energy into
offering a Linux that is guaranteed free of OOM surprises.

So, let me repeat the RFC.
Apply the above patch, and do "echo 2 > /proc/sys/vm/overcommit_memory".
Now test. In case you have no, or only a small amount of swap space,
also do "echo 80 > /proc/sys/vm/overcommit_ratio" or so.

Andries
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer ...

[Marcelo Tosatti]

Hi Andries,

On Sat, Nov 06, 2004 at 01:53:17PM +0100, Andries Brouwer wrote:
> On Fri, Nov 05, 2004 at 06:01:18PM -0200, Marcelo Tosatti wrote:
> 
> > My wife is almost killing me, its Friday night and I've been telling her
> > "just another minute" for hours. Have to run.
> 
> :-)
> 
> > As you know the OOM is very problematic in 2.6 right now - so I went
> > to investigate it.
> 
> I have always been surprised that so few people investigated
> doing things right, that is, entirely without OOM killer.
> Apparently developers do not think about using Linux for serious work
> where it can be a disaster, possibly even a life-threatening disaster,
> when any process can be killed at any time.

Its just that the majority of users use total overcommit (the default), 
but you have a point.

> Ten years ago it was a bad waste of resources to have swapspace
> lying around that would be used essentially 0% of the time.
> But with todays disk sizes it is entirely feasible to have
> a few hundred MB of "unused" swap space. A small price to
> pay for the guarantee that no process will be OOM killed.
> 
> A month ago I showed a patch that made overcommit mode 2
> work for me. Google finds it in http://lwn.net/Articles/104959/
> 
> So far, nobody commented.
> 
> This is not in a state such that I would like to submit it,
> but I think it would be good to focus some energy into
> offering a Linux that is guaranteed free of OOM surprises.

I dont have any useful comments on patch on a quick look at it  - 
but yes non-overcommit should be working correctly.

> So, let me repeat the RFC.
> Apply the above patch, and do "echo 2 > /proc/sys/vm/overcommit_memory".
> Now test. In case you have no, or only a small amount of swap space,
> also do "echo 80 > /proc/sys/vm/overcommit_ratio" or so.

Will test your patch later on the weekend and take a slower look 
at it, hopefully with useful comments.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer ...

[Marko Macek]

Andries Brouwer wrote:

> I have always been surprised that so few people investigated
> doing things right, that is, entirely without OOM killer.

Agreed.

> This is not in a state such that I would like to submit it,
> but I think it would be good to focus some energy into
> offering a Linux that is guaranteed free of OOM surprises.

A good thing would be to make the OOM killer only kill
processes that actually overcommit (independant of overcommit mode).

The first step would be adding a value in /proc/$pid/...
somewhere that shows how much a process is overcommitted when
overcommit is enabled. This would allow important processes to be
fixed for all overcommit modes.


	MArk
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] Remove OOM killer from try_to_free_pages / all_unreclaimable braindamage

[Marcelo Tosatti]

On Fri, Nov 05, 2004 at 06:01:18PM -0200, Marcelo Tosatti wrote:

> While doing this, I noticed that kswapd will happily go to sleep 
> if all zones have all_unreclaimable set. I bet this is the reason 
> for the page allocation failures we are seeing. So the patch 
> also makes balance_pgdat() NOT return and go to "loop_again" 
> instead in case of page shortage - even if all_unreclaimable is set.
> 
> Basically the "loop_again" logic IS NOT WORKING! 

Wrong, the loop_again logic is working, all_zones_ok will be
set when DEF_PRIORITY = 0. 

So the page allocation failures are happening for some other 
reason(s).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>