Re: throttling dirtiers

[William Lee Irwin III <wli@holomorphy.com>]

On Wed, Jul 31, 2002 at 01:26:09AM -0700, Andrew Morton wrote:
> Here's an interesting test.
> - mem=512m
> - Run a program which mallocs 400 megs and then madly sits
>   there touching each page.
> - Do a big `dd' to a file.
> Everything works nicely - all the anon memory sits on the active
> list, all writeback is via shrink_cache -> vm_writeback.
> Bandwidth is good.

The VM has to do some because pages can be dirtied by mmap()'d access.
Only scanning for modified bits or trapping write access (ugh) can find
these pages.


On Wed, Jul 31, 2002 at 01:26:09AM -0700, Andrew Morton wrote:
> But as we discussed, we really shouldn't be doing the IO from
> within the VM.  balance_dirty_pages() is never triggering
> because the system is not reaching 40% dirty.
> It would make sense for the VM to detect an overload
> of dirty pages coming off the tail of the LRU and to reach
> over and tell balance_dirty_pages() to provide throttling,
> If we were to say "gee, of the last 1,000 pages, 25% were
> dirty, so tell balance_dirty_pages() to throttle everyone"
> then that would be too late because the LRU will be _full_
> of dirty pages.

balance_dirty_pages() is the closest thing to source throttling
available, so it should definitely be used before VM background
writeback. Perhaps assigning dirty page budgets to tasks and/or
struct address_space and checking for budget excess would be good?
Trouble is I'm not sure exactly how well they can be enforced
given the mmap() problem.


On Wed, Jul 31, 2002 at 01:26:09AM -0700, Andrew Morton wrote:
> I can't think of a sane way of keeping track of the number
> of dirty pages on the inactive list, because the locking
> is quite disjoint.
> But we can certainly track the amount of anon memory in
> the machine, and set the balance_dirty_pages thresholds
> at 0.4 * (total memory - anon memory) or something like
> that.
> Thoughts?

I'm not a fan of this kind of global decision. For example, I/O devices
may be fast enough and memory small enough to dump all memory in < 1s,
in which case dirtying most or all of memory is okay from a latency
standpoint, or it may take hours to finish dumping out 40% of memory,
in which case it should be far more eager about writeback.


Cheers,
Bill
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/