bdflush defaults bugreport

bdflush defaults bugreport

[Rik van Riel]

Hi,

yesterday evening I've seen a 32MB machine failing to install because
mke2fs was killed due to memory shortage -- memory shortage due to
a too large number of dirty blocks (max 40% by default).

Lowering the number to 1% solved all problems, so I guess we should
lower the number in the kernel to something like 10%, which should
be _more_ than enough since the page cache can now be dirty too...

Btw, the problem happened on a 2.2.10 machine, so I guess we should
lower the 2.2 default as well (to 15%? 20%?).

regards,

Rik
--
The Internet is not a network of computers. It is a network
of people. That is its real strength.
--
work at:	http://www.reseau.nl/
home at:	http://www.nl.linux.org/~riel/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/

[2.2.12 PATCH] Re: bdflush defaults bugreport

[Benjamin C.R. LaHaise]

Hey Rik,

> yesterday evening I've seen a 32MB machine failing to install because
> mke2fs was killed due to memory shortage -- memory shortage due to
> a too large number of dirty blocks (max 40% by default).
> 
> Lowering the number to 1% solved all problems, so I guess we should
> lower the number in the kernel to something like 10%, which should
> be _more_ than enough since the page cache can now be dirty too...
> 
> Btw, the problem happened on a 2.2.10 machine, so I guess we should
> lower the 2.2 default as well (to 15%? 20%?).

I don't quite think that changing the percentage dirty is the right thing
in this case.  Rather, the semantics of refile_buffer / wakeup_bdflush /
mark_buffer_clean need to be tweaked: as it stands, bdflush will wake
bdflush_done before the percentage of dirty buffers drops below the
threshhold.  The right fix should be to move the wake_up into the if
checking the threshhold right below it as the only user of bdflush_done is
from wake_bdflush when too many buffers are dirty.  Patch below (albeit
untested).  Alan/Stephen: comments?

		-ben

--- buffer.c.orig	Mon Aug  9 15:04:40 1999
+++ buffer.c	Sun Sep  5 04:35:06 1999
@@ -1813,11 +1813,12 @@
 			continue;
 		}
 		run_task_queue(&tq_disk);
-		wake_up(&bdflush_done);
 		
 		/* If there are still a lot of dirty buffers around, skip the sleep
 		   and flush some more */
 		if(ndirty == 0 || nr_buffers_type[BUF_DIRTY] <= nr_buffers * bdf_prm.b_un.nfract/100) {
+			wake_up(&bdflush_done);
+
 			spin_lock_irq(&current->sigmask_lock);
 			flush_signals(current);
 			spin_unlock_irq(&current->sigmask_lock);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/

Re: bdflush defaults bugreport

[Stephen C. Tweedie]

Hi,

On Sun, 5 Sep 1999 09:58:56 +0200 (CEST), Rik van Riel
<riel@humbolt.geo.uu.nl> said:

> yesterday evening I've seen a 32MB machine failing to install because
> mke2fs was killed due to memory shortage -- memory shortage due to
> a too large number of dirty blocks (max 40% by default).

In the past, such problems were mainly due to the refile_buffer() code
not limiting the write rate of heavy buffer cache writes more than
anything else.

> Lowering the number to 1% solved all problems, so I guess we should
> lower the number in the kernel to something like 10%, which should
> be _more_ than enough since the page cache can now be dirty too...

That is not a clean solution: it's just imposing an unnecessary
performance on the whole cache when the problem probably lies
elsewhere.  

> Btw, the problem happened on a 2.2.10 machine

In 2.2, the page cache cannot be dirty in this sense.

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/

Re: [2.2.12 PATCH] Re: bdflush defaults bugreport

[Stephen C. Tweedie]

Hi,

On Sun, 5 Sep 1999 04:55:41 -0400 (EDT), "Benjamin C.R. LaHaise"
<blah@kvack.org> said:

> I don't quite think that changing the percentage dirty is the right thing
> in this case.  Rather, the semantics of refile_buffer / wakeup_bdflush /
> mark_buffer_clean need to be tweaked: as it stands, bdflush will wake
> bdflush_done before the percentage of dirty buffers drops below the
> threshhold.  The right fix should be to move the wake_up into the if
> checking the threshhold right below it as the only user of bdflush_done is
> from wake_bdflush when too many buffers are dirty.  Patch below (albeit
> untested).  Alan/Stephen: comments?

It shouldn't have any noticeable effect: all that would happen with that
patch applied is that the woken process would wake early, write one more
buffer and drop straight back into the wakeup_bdflush(1) stall.

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/

Re: bdflush defaults bugreport

[Stephen C. Tweedie]

Hi,

On Mon, 13 Sep 1999 14:11:36 -0400 (EDT), "Benjamin C.R. LaHaise"
<blah@kvack.org> said:

> I'm not quite sure if you caught my original response to Rik, but the
> problem seems to stem from the fact that bdflush is waking users without
> checking if the % dirty buffers is low enough.  Just moving the wakeup a
> couple of lines down looks like a solution (although I can't say for
> certain that the performance effects won't be dreadful).

I'm not sure if you caught my other response, but moving that wakeup one
line down will just result in the writing task being able to write one
more block before blocking on bdflush yet again.

> There's another problem with 2.2.10+: by limiting to the percentage of
> dirty buffers, behaviour on temp files/with lots of free ram is ugly.  I'm
> wondering if limiting according to % memory dirty instead would be
> reasonable -- your thoughts?

Close.  The real solution is probably to reserve a %age of memory to be
clean.  In other words the limit shouldn't just be on dirty memory, it
should be on dirty pages plus non-movable pages (kmalloc etc).  

--Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://humbolt.geo.uu.nl/Linux-MM/