Re: [RFC][PATCH 1/1] Expose per-node reclaim and migration to userspace

["Paul Menage" <menage@google.com>]

On 11/30/06, Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> >> AFAIK they do that in their higher level APIs (at least HPC numa does).
> >
> >
> > Could you point me at an example?
>
> kernel/cpuset.c:cpuset_migrate_mm

No, that doesn't really do what we want. It basically just calls
do_migrate_pages, which has the drawbacks of:

- it has no way to try to migrate memory from one source node to
multiple destination nodes.

- it doesn't (as far as I can tell) migrate unmapped file pages in the
page cache.

- it scans every page table entry of every mm in the process. If your
nodes are relatively small compared to your processes, this is likely
to be much more heavyweight than just trying to migrate each page in a
node. (I realise that there are some unsolved implementation issues
with migrating pages whilst not holding an mmap_sem of an mm that's
mapping them; that's something that we would need to solve)

>
> How about "try to change the memory reservation charge of this
> 'container' from xMB to yMB"? Underneath that API, your fakenode
> controller would do the node reclaim and consolidation stuff --
> but it could be implemented completely differently in the case of
> a different type of controller.

How would it make decisions such as which node to free up (e.g.
userspace might have a strong preference for keeping a job on one
particular real node, or moving it to a different one.) I think that
policy decisions like this belong in userspace, in the same way that
the existing cpusets API provides a way to say "this cpuset uses these
nodes" rather than "this cpuset should have N nodes".

If the API was expressive enough to say "try to shrink this cpuset by
X MB, with amount Y of effort, trying to evict nodes in the priority
order A,B,C" that might be a good start.

>
> >> The cpusets code is definitely similar to what memory resource control
> >> needs. I don't think that a resource control API needs to be tied to
> >> such granular, hard limits as the fakenodes code provides though. But
> >> maybe I'm wrong and it really would be acceptable for everyone.
> >
> >
> > Ah. This isn't intended to be specifically a "resource control API".
> > It's more intended to be an API that could be useful for certain kinds
> > of resource control, but could also be generically useful.
>
> If it is exporting any kind of implementation details, then it needs
> to be justified with a specific user that can't be implemented in a
> better way, IMO.

It's not really exporting any more implementation details than the
existing cpusets API (i.e. explicitly binding a job to a set of nodes
chosen by userspace). The only true exposed implementation detail is
the "priority" value from try_to_free_pages, and that could be
abstracted away as a value in some range 0-N where 0 means "try very
hard" and N means "hardly try at all", and it wouldn't have to be
directly linked to the try_to_free_pages() priority.

Paul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>