Re: [patch 10/12] Memoryless nodes: Update memory policy and page migration

Re: [patch 10/12] Memoryless nodes: Update memory policy and page migration

[KAMEZAWA Hiroyuki]

On Tue, 10 Jul 2007 14:52:15 -0700
Christoph Lameter <clameter@sgi.com> wrote:

> +			*nodes = node_memory_map;
>  		else
node_states[N_MEMORY]  ?


>  		check_pgd_range(vma, vma->vm_start, vma->vm_end,
> -				&node_online_map, MPOL_MF_STATS, md);
> +				&node_memory_map, MPOL_MF_STATS, md);
>  	}

Again here.

-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 11/12] Memoryless nodes: Fix GFP_THISNODE behavior

[KAMEZAWA Hiroyuki]

some nitpicks...

On Tue, 10 Jul 2007 14:52:16 -0700
Christoph Lameter <clameter@sgi.com> wrote:

> +static void build_thisnode_zonelists(pg_data_t *pgdat)
> +{
> +	enum zone_type i;
> +	int j;
> +	struct zonelist *zonelist;
> +
> +	for (i = 0; i < MAX_NR_ZONES; i++) {
> +		zonelist = pgdat->node_zonelists + MAX_NR_ZONES + i;
> + 		j = build_zonelists_node(pgdat, zonelist, 0, i);
> +		zonelist->zones[j] = NULL;
> +	}
> +}
adding explanation as following is maybe good.
==
	/* 
         * NUMA default zonelist is structured as
	 * [0....MAX_NR_ZONES) : allows fallbacks to other node for each GFP_MASK.
	 * [MAX_NR_ZONES...MAX_ZONELISTS) : disallow fallbacks for GFP_XXX |GFP_THISNODE
	 */
==
> +
> +/*
>   * Build zonelists ordered by zone and nodes within zones.
>   * This results in conserving DMA zone[s] until all Normal memory is
>   * exhausted, but results in overflowing to remote node while memory
> @@ -2267,7 +2283,7 @@ static void build_zonelists(pg_data_t *p
>  	int order = current_zonelist_order;
>  
>  	/* initialize zonelists */
> -	for (i = 0; i < MAX_NR_ZONES; i++) {
> +	for (i = 0; i < 2 * MAX_NR_ZONES; i++) {

please use MAX_ZONELISTS here.

-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 08/12] Uncached allocator: Handle memoryless nodes

[Jes Sorensen]

Christoph Lameter wrote:
> The checks for node_online in the uncached allocator are made to make sure
> that memory is available on these nodes. Thus switch all the checks to use
> the node_memory and for_each_memory_node functions.
> 
> Cc: jes@sgi.com
> Signed-off-by: Christoph Lameter <clameter@sgi.com>

I'm fine with this:

Signed-off-by: Jes Sorensen <jes@sgi.com>

Jes


> ---
>  arch/ia64/kernel/uncached.c |    4 ++--
>  drivers/char/mspec.c        |    2 +-
>  2 files changed, 3 insertions(+), 3 deletions(-)
> 
> Index: linux-2.6.22-rc6-mm1/arch/ia64/kernel/uncached.c
> ===================================================================
> --- linux-2.6.22-rc6-mm1.orig/arch/ia64/kernel/uncached.c	2007-06-24 16:21:48.000000000 -0700
> +++ linux-2.6.22-rc6-mm1/arch/ia64/kernel/uncached.c	2007-07-09 22:24:42.000000000 -0700
> @@ -196,7 +196,7 @@ unsigned long uncached_alloc_page(int st
>  	nid = starting_nid;
>  
>  	do {
> -		if (!node_online(nid))
> +		if (!node_state(nid, N_MEMORY))
>  			continue;
>  		uc_pool = &uncached_pools[nid];
>  		if (uc_pool->pool == NULL)
> @@ -268,7 +268,7 @@ static int __init uncached_init(void)
>  {
>  	int nid;
>  
> -	for_each_online_node(nid) {
> +	for_each_node_state(nid, N_ONLINE) {
>  		uncached_pools[nid].pool = gen_pool_create(PAGE_SHIFT, nid);
>  		mutex_init(&uncached_pools[nid].add_chunk_mutex);
>  	}
> Index: linux-2.6.22-rc6-mm1/drivers/char/mspec.c
> ===================================================================
> --- linux-2.6.22-rc6-mm1.orig/drivers/char/mspec.c	2007-07-03 17:19:24.000000000 -0700
> +++ linux-2.6.22-rc6-mm1/drivers/char/mspec.c	2007-07-09 22:24:42.000000000 -0700
> @@ -353,7 +353,7 @@ mspec_init(void)
>  		is_sn2 = 1;
>  		if (is_shub2()) {
>  			ret = -ENOMEM;
> -			for_each_online_node(nid) {
> +			for_each_node_state(nid, N_ONLINE) {
>  				int actual_nid;
>  				int nasid;
>  				unsigned long phys;
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 10/12] Memoryless nodes: Update memory policy and page migration

[Nishanth Aravamudan]

On 11.07.2007 [16:48:11 +0900], KAMEZAWA Hiroyuki wrote:
> On Tue, 10 Jul 2007 14:52:15 -0700
> Christoph Lameter <clameter@sgi.com> wrote:
> 
> > +			*nodes = node_memory_map;
> >  		else
> node_states[N_MEMORY]  ?
> 
> 
> >  		check_pgd_range(vma, vma->vm_start, vma->vm_end,
> > -				&node_online_map, MPOL_MF_STATS, md);
> > +				&node_memory_map, MPOL_MF_STATS, md);
> >  	}
> 
> Again here.

I think Christoph missed a hunk in the node_memory_map patch, which
would

#define node_memory_map node_stats[N_MEMORY]

Thanks,
Nish

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 02/12] NUMA: Introduce node_memory_map

[Nishanth Aravamudan]

On 10.07.2007 [14:52:07 -0700], Christoph Lameter wrote:
> It is necessary to know if nodes have memory since we have recently
> begun to add support for memoryless nodes. For that purpose we introduce
> a new node state N_MEMORY.
> 
> A node has its bit in node_memory_map set if it has memory. If a node
> has memory then it has at least one zone defined in its pgdat structure
> that is located in the pgdat itself.

Uh, except node_memory_map is not defined below.

I'm guessing you just need

#define	node_memory_map	node_states[N_MEMORY]

below.

Thanks,
Nish

> N_MEMORY can then be used in various places to insure that we
> do the right thing when we encounter a memoryless node.
> 
> Signed-off-by: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
> Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
> Signed-off-by: Christoph Lameter <clameter@sgi.com>
> 
> ---
>  include/linux/nodemask.h |    1 +
>  mm/page_alloc.c          |    9 +++++++--
>  2 files changed, 8 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6.22-rc6-mm1/include/linux/nodemask.h
> ===================================================================
> --- linux-2.6.22-rc6-mm1.orig/include/linux/nodemask.h	2007-07-09 22:13:44.000000000 -0700
> +++ linux-2.6.22-rc6-mm1/include/linux/nodemask.h	2007-07-09 22:16:05.000000000 -0700
> @@ -343,6 +343,7 @@ static inline void __nodes_remap(nodemas
>  enum node_states {
>  	N_POSSIBLE,	/* The node could become online at some point */
>  	N_ONLINE,	/* The node is online */
> +	N_MEMORY,	/* The node has memory */
>  	NR_NODE_STATES
>  };
> 
> Index: linux-2.6.22-rc6-mm1/mm/page_alloc.c
> ===================================================================
> --- linux-2.6.22-rc6-mm1.orig/mm/page_alloc.c	2007-07-09 22:15:45.000000000 -0700
> +++ linux-2.6.22-rc6-mm1/mm/page_alloc.c	2007-07-09 22:19:28.000000000 -0700
> @@ -2392,8 +2392,13 @@ static int __build_all_zonelists(void *d
>  	int nid;
> 
>  	for_each_online_node(nid) {
> -		build_zonelists(NODE_DATA(nid));
> -		build_zonelist_cache(NODE_DATA(nid));
> +		pg_data_t *pgdat = NODE_DATA(nid);
> +
> +		build_zonelists(pgdat);
> +		build_zonelist_cache(pgdat);
> +
> +		if (pgdat->node_present_pages)
> +			node_set_state(nid, N_MEMORY);
>  	}
>  	return 0;
>  }
> 
> -- 

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 10/12] Memoryless nodes: Update memory policy and page migration

[Nishanth Aravamudan]

On 11.07.2007 [09:16:53 -0700], Nishanth Aravamudan wrote:
> On 11.07.2007 [16:48:11 +0900], KAMEZAWA Hiroyuki wrote:
> > On Tue, 10 Jul 2007 14:52:15 -0700
> > Christoph Lameter <clameter@sgi.com> wrote:
> > 
> > > +			*nodes = node_memory_map;
> > >  		else
> > node_states[N_MEMORY]  ?
> > 
> > 
> > >  		check_pgd_range(vma, vma->vm_start, vma->vm_end,
> > > -				&node_online_map, MPOL_MF_STATS, md);
> > > +				&node_memory_map, MPOL_MF_STATS, md);
> > >  	}
> > 
> > Again here.
> 
> I think Christoph missed a hunk in the node_memory_map patch, which
> would
> 
> #define node_memory_map node_stats[N_MEMORY]

Or maybe it's intentional -- and your changes are appropriate (and
in-line with the other changes Christoph's patches made).

Thanks,
Nish

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 10/12] Memoryless nodes: Update memory policy and page migration

[Christoph Lameter]

On Wed, 11 Jul 2007, Nishanth Aravamudan wrote:

> I think Christoph missed a hunk in the node_memory_map patch, which
> would
> 
> #define node_memory_map node_stats[N_MEMORY]

No. Somehow the patch was not updated. Need to send out a new rev.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 11/12] Memoryless nodes: Fix GFP_THISNODE behavior

[Christoph Lameter]

On Wed, 11 Jul 2007, KAMEZAWA Hiroyuki wrote:

> > +	struct zonelist *zonelist;
> > +
> > +	for (i = 0; i < MAX_NR_ZONES; i++) {
> > +		zonelist = pgdat->node_zonelists + MAX_NR_ZONES + i;
> > + 		j = build_zonelists_node(pgdat, zonelist, 0, i);
> > +		zonelist->zones[j] = NULL;
> > +	}
> > +}
> adding explanation as following is maybe good.
> ==
> 	/* 
>          * NUMA default zonelist is structured as
> 	 * [0....MAX_NR_ZONES) : allows fallbacks to other node for each GFP_MASK.
> 	 * [MAX_NR_ZONES...MAX_ZONELISTS) : disallow fallbacks for GFP_XXX |GFP_THISNODE
> 	 */
> ==


Will add this.
> > +
> > +/*
> >   * Build zonelists ordered by zone and nodes within zones.
> >   * This results in conserving DMA zone[s] until all Normal memory is
> >   * exhausted, but results in overflowing to remote node while memory
> > @@ -2267,7 +2283,7 @@ static void build_zonelists(pg_data_t *p
> >  	int order = current_zonelist_order;
> >  
> >  	/* initialize zonelists */
> > -	for (i = 0; i < MAX_NR_ZONES; i++) {
> > +	for (i = 0; i < 2 * MAX_NR_ZONES; i++) {
> 
> please use MAX_ZONELISTS here.

Right. Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 10/12] Memoryless nodes: Update memory policy and page migration

[Christoph Lameter]

On Wed, 11 Jul 2007, Nishanth Aravamudan wrote:

> > Index: linux-2.6.22-rc6-mm1/mm/migrate.c
> > ===================================================================
> > --- linux-2.6.22-rc6-mm1.orig/mm/migrate.c	2007-07-09 21:23:18.000000000 -0700
> > +++ linux-2.6.22-rc6-mm1/mm/migrate.c	2007-07-11 10:37:03.000000000 -0700
> > @@ -963,7 +963,7 @@ asmlinkage long sys_move_pages(pid_t pid
> >  				goto out;
> > 
> >  			err = -ENODEV;
> > -			if (!node_online(node))
> > +			if (!node_memory(node))
> 
> 			if (!node_state(node, N_MEMORY))
> 
> ?

Next patch fixes it up: :-=

Fixed up version


Memoryless nodes: Update memory policy and page migration

Online nodes now may have no memory. The checks and initialization must therefore
be changed to no longer use the online functions.

This will correctly initialize the interleave on bootup to only target
nodes with memory and will make sys_move_pages return an error when a page
is to be moved to a memoryless node. Similarly we will get an error if
MPOL_BIND and MPOL_INTERLEAVE is used on a memoryless node.

These are somewhat new semantics. So far one could specify memoryless nodes
and we would maybe do the right thing and just ignore the node (or we'd do
something strange like with MPOL_INTERLEAVE). If we want to allow the
specification of memoryless nodes via memory policies then we need to keep
checking for online nodes.

Signed-off-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Nishanth Aravamudan <nacc@us.ibm.com>

---
 mm/mempolicy.c |   10 +++++-----
 mm/migrate.c   |    2 +-
 2 files changed, 6 insertions(+), 6 deletions(-)

Index: linux-2.6.22-rc6-mm1/mm/migrate.c
===================================================================
--- linux-2.6.22-rc6-mm1.orig/mm/migrate.c	2007-07-11 11:49:33.000000000 -0700
+++ linux-2.6.22-rc6-mm1/mm/migrate.c	2007-07-11 11:51:34.000000000 -0700
@@ -963,7 +963,7 @@ asmlinkage long sys_move_pages(pid_t pid
 				goto out;
 
 			err = -ENODEV;
-			if (!node_online(node))
+			if (!node_state(node, N_MEMORY))
 				goto out;
 
 			err = -EACCES;
Index: linux-2.6.22-rc6-mm1/mm/mempolicy.c
===================================================================
--- linux-2.6.22-rc6-mm1.orig/mm/mempolicy.c	2007-07-11 11:49:39.000000000 -0700
+++ linux-2.6.22-rc6-mm1/mm/mempolicy.c	2007-07-11 11:49:48.000000000 -0700
@@ -496,9 +496,9 @@ static void get_zonemask(struct mempolic
 		*nodes = p->v.nodes;
 		break;
 	case MPOL_PREFERRED:
-		/* or use current node instead of online map? */
+		/* or use current node instead of memory_map? */
 		if (p->v.preferred_node < 0)
-			*nodes = node_online_map;
+			*nodes = node_states[N_MEMORY];
 		else
 			node_set(p->v.preferred_node, *nodes);
 		break;
@@ -1618,7 +1618,7 @@ void __init numa_policy_init(void)
 	 * fall back to the largest node if they're all smaller.
 	 */
 	nodes_clear(interleave_nodes);
-	for_each_online_node(nid) {
+	for_each_node_state(nid, N_MEMORY) {
 		unsigned long total_pages = node_present_pages(nid);
 
 		/* Preserve the largest node */
@@ -1898,7 +1898,7 @@ int show_numa_map(struct seq_file *m, vo
 		seq_printf(m, " huge");
 	} else {
 		check_pgd_range(vma, vma->vm_start, vma->vm_end,
-				&node_online_map, MPOL_MF_STATS, md);
+				&node_states[N_MEMORY], MPOL_MF_STATS, md);
 	}
 
 	if (!md->pages)
@@ -1925,7 +1925,7 @@ int show_numa_map(struct seq_file *m, vo
 	if (md->writeback)
 		seq_printf(m," writeback=%lu", md->writeback);
 
-	for_each_online_node(n)
+	for_each_node_state(n, N_MEMORY)
 		if (md->node[n])
 			seq_printf(m, " N%d=%lu", n, md->node[n]);
 out:

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 10/12] Memoryless nodes: Update memory policy and page migration

[Nishanth Aravamudan]

On 11.07.2007 [11:22:29 -0700], Christoph Lameter wrote:
> Online nodes now may have no memory. The checks and initialization must therefore
> be changed to no longer use the online functions.
> 
> This will correctly initialize the interleave on bootup to only target
> nodes with memory and will make sys_move_pages return an error when a page
> is to be moved to a memoryless node. Similarly we will get an error if
> MPOL_BIND and MPOL_INTERLEAVE is used on a memoryless node.
> 
> These are somewhat new semantics. So far one could specify memoryless nodes
> and we would maybe do the right thing and just ignore the node (or we'd do
> something strange like with MPOL_INTERLEAVE). If we want to allow the
> specification of memoryless nodes via memory policies then we need to keep
> checking for online nodes.
> 
> Signed-off-by: Christoph Lameter <clameter@sgi.com>
> Acked-by: Nishanth Aravamudan <nacc@us.ibm.com>
> 
> ---
>  mm/mempolicy.c |   10 +++++-----
>  mm/migrate.c   |    2 +-
>  2 files changed, 6 insertions(+), 6 deletions(-)
> 
> Index: linux-2.6.22-rc6-mm1/mm/migrate.c
> ===================================================================
> --- linux-2.6.22-rc6-mm1.orig/mm/migrate.c	2007-07-09 21:23:18.000000000 -0700
> +++ linux-2.6.22-rc6-mm1/mm/migrate.c	2007-07-11 10:37:03.000000000 -0700
> @@ -963,7 +963,7 @@ asmlinkage long sys_move_pages(pid_t pid
>  				goto out;
> 
>  			err = -ENODEV;
> -			if (!node_online(node))
> +			if (!node_memory(node))

			if (!node_state(node, N_MEMORY))

?

Thanks,
Nish

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>