Re: [patch 00/14] NUMA: Memoryless node support V3

Re: [patch 00/14] NUMA: Memoryless node support V3

[Nishanth Aravamudan]

On 03.08.2007 [20:01:00 -0700], Christoph Lameter wrote:
> V4->V5
> - Split N_MEMORY into N_NORMAL_MEMORY and N_HIGH_MEMORY to support
>   32 bit NUMA.
> - Mel tested it on 32bit NUMA
> - !NUMA Fixes
> - Tested on SMP and UP.
> 
> V3->V4 (by Lee)
> - Add fixes and testing.
> 
> V2->V3
> - Refresh patches (sigh)
> - Add comments suggested by Kamezawa Hiroyuki
> - Add signoff by Jes Sorensen
> 
> V1->V2
> - Add a generic layer that allows the definition of additional node bitmaps
> 
> This patchset is implementing additional node bitmaps that allow the system
> to track nodes that are online without memory and nodes that have processors.
> 
> Note that this patch is only the beginning. All code portions that assume that
> an online node has memory must be changed to use either N_NORMAL_MEMORY or
> N_HIGH_MEMORY.

I believe Andrew should drop the 9 memoryless node patches he had picked
up and take these instead. The NORMAL_MEMORY/HIGH_MEMORY distinction is
critical for booting 32-bit NUMA. I've rebased my stack of hugetlb
related patches on top of these and am testing now.

Thanks,
Nish

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 02/14] Memoryless nodes: introduce mask of nodes with memory

[Andrew Morton]

On Fri, 03 Aug 2007 20:01:02 -0700
Christoph Lameter <clameter@sgi.com> wrote:

> +/* Any regular memory on that node ? */
> +static void check_for_regular_memory(pg_data_t *pgdat)
> +{
> +#ifdef CONFIG_HIGHMEM
> +	enum zone_type zone;
> +
> +	for (zone = 0; zone <= ZONE_NORMAL; zone++)
> +		if (pgdat->node_zones[zone].present_pages)
> +			node_set_state(nid, N_NORMAL_MEMORY);
> +	}
> +#endif
> +}

mm/page_alloc.c: In function 'check_for_regular_memory':
mm/page_alloc.c:2427: error: 'nid' undeclared (first use in this function)
mm/page_alloc.c:2427: error: (Each undeclared identifier is reported only once
mm/page_alloc.c:2427: error: for each function it appears in.)
mm/page_alloc.c: At top level:
mm/page_alloc.c:2430: error: expected identifier or '(' before '}' token

OK, easily fixable with

/* Any regular memory on that node ? */
static void check_for_regular_memory(pg_data_t *pgdat)
{
#ifdef CONFIG_HIGHMEM
	enum zone_type zone_type;
	
	for (zone_type = 0; zone_type <= ZONE_NORMAL; zone_type++) {
		struct zone *zone = &pgdat->node_zones[zone_type];
		if (zone->present_pages)
			node_set_state(zone_to_nid(zone), N_NORMAL_MEMORY);
	}
#endif
}

but we continue to have some significant testing issues out there.		

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 02/14] Memoryless nodes: introduce mask of nodes with memory

[Nishanth Aravamudan]

On 08.08.2007 [12:38:04 -0700], Andrew Morton wrote:
> On Fri, 03 Aug 2007 20:01:02 -0700
> Christoph Lameter <clameter@sgi.com> wrote:
> 
> > +/* Any regular memory on that node ? */
> > +static void check_for_regular_memory(pg_data_t *pgdat)
> > +{
> > +#ifdef CONFIG_HIGHMEM
> > +	enum zone_type zone;
> > +
> > +	for (zone = 0; zone <= ZONE_NORMAL; zone++)
> > +		if (pgdat->node_zones[zone].present_pages)
> > +			node_set_state(nid, N_NORMAL_MEMORY);
> > +	}
> > +#endif
> > +}
> 
> mm/page_alloc.c: In function 'check_for_regular_memory':
> mm/page_alloc.c:2427: error: 'nid' undeclared (first use in this function)
> mm/page_alloc.c:2427: error: (Each undeclared identifier is reported only once
> mm/page_alloc.c:2427: error: for each function it appears in.)
> mm/page_alloc.c: At top level:
> mm/page_alloc.c:2430: error: expected identifier or '(' before '}' token
> 
> OK, easily fixable with
> 
> /* Any regular memory on that node ? */
> static void check_for_regular_memory(pg_data_t *pgdat)
> {
> #ifdef CONFIG_HIGHMEM
> 	enum zone_type zone_type;
> 	
> 	for (zone_type = 0; zone_type <= ZONE_NORMAL; zone_type++) {
> 		struct zone *zone = &pgdat->node_zones[zone_type];
> 		if (zone->present_pages)
> 			node_set_state(zone_to_nid(zone), N_NORMAL_MEMORY);
> 	}
> #endif
> }
> 
> but we continue to have some significant testing issues out there.		

This would be because the patch that Christoph submitted here is not the
same as the patch that Mel and I tested...There was no
check_for_regular_memory() function in the kernels I was building.

Thanks,
Nish

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 02/14] Memoryless nodes: introduce mask of nodes with memory

[Nishanth Aravamudan]

On 08.08.2007 [12:55:14 -0700], Nishanth Aravamudan wrote:
> On 08.08.2007 [12:38:04 -0700], Andrew Morton wrote:
> > On Fri, 03 Aug 2007 20:01:02 -0700
> > Christoph Lameter <clameter@sgi.com> wrote:
> > 
> > > +/* Any regular memory on that node ? */
> > > +static void check_for_regular_memory(pg_data_t *pgdat)
> > > +{
> > > +#ifdef CONFIG_HIGHMEM
> > > +	enum zone_type zone;
> > > +
> > > +	for (zone = 0; zone <= ZONE_NORMAL; zone++)
> > > +		if (pgdat->node_zones[zone].present_pages)
> > > +			node_set_state(nid, N_NORMAL_MEMORY);
> > > +	}
> > > +#endif
> > > +}
> > 
> > mm/page_alloc.c: In function 'check_for_regular_memory':
> > mm/page_alloc.c:2427: error: 'nid' undeclared (first use in this function)
> > mm/page_alloc.c:2427: error: (Each undeclared identifier is reported only once
> > mm/page_alloc.c:2427: error: for each function it appears in.)
> > mm/page_alloc.c: At top level:
> > mm/page_alloc.c:2430: error: expected identifier or '(' before '}' token
> > 
> > OK, easily fixable with
> > 
> > /* Any regular memory on that node ? */
> > static void check_for_regular_memory(pg_data_t *pgdat)
> > {
> > #ifdef CONFIG_HIGHMEM
> > 	enum zone_type zone_type;
> > 	
> > 	for (zone_type = 0; zone_type <= ZONE_NORMAL; zone_type++) {
> > 		struct zone *zone = &pgdat->node_zones[zone_type];
> > 		if (zone->present_pages)
> > 			node_set_state(zone_to_nid(zone), N_NORMAL_MEMORY);
> > 	}
> > #endif
> > }
> > 
> > but we continue to have some significant testing issues out there.		
> 
> This would be because the patch that Christoph submitted here is not the
> same as the patch that Mel and I tested...There was no
> check_for_regular_memory() function in the kernels I was building.

To try and remedy this -- I'll regrab this stack and rebase my patches
again. Test everything and resubmit mine.

Thanks,
Nish

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 02/14] Memoryless nodes: introduce mask of nodes with memory

[Christoph Lameter]

On Wed, 8 Aug 2007, Nishanth Aravamudan wrote:

> To try and remedy this -- I'll regrab this stack and rebase my patches
> again. Test everything and resubmit mine.

Ummmm... These are the patches that you said you would test earlier.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 02/14] Memoryless nodes: introduce mask of nodes with memory

[Nishanth Aravamudan]

On 08.08.2007 [13:05:12 -0700], Christoph Lameter wrote:
> On Wed, 8 Aug 2007, Nishanth Aravamudan wrote:
> 
> > To try and remedy this -- I'll regrab this stack and rebase my patches
> > again. Test everything and resubmit mine.
> 
> Ummmm... These are the patches that you said you would test earlier.

Yes, sorry, I had leafed through the patches and didn't see any changes
and didn't see any subject changes indicated UPDATED or a new version,
so I incorrectly assumed that the patches were the same as before.

This stack has now been successfully compile & boot-tested on:

4-node x86 (NUMAQ), 1-node x86 (NUMAQ), !NUMA x86, 2-node IA64, 4-node
ppc64 (2 memoryless nodes).

Thanks,
Nish

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 02/14] Memoryless nodes: introduce mask of nodes with memory

[Christoph Lameter]

On Wed, 8 Aug 2007, Nishanth Aravamudan wrote:

> > but we continue to have some significant testing issues out there.		
> 
> This would be because the patch that Christoph submitted here is not the
> same as the patch that Mel and I tested...There was no
> check_for_regular_memory() function in the kernels I was building.

Right. I added that to #ifdef out the HIGHMEM/NORMAL distinction for 
regular NUMA boxes. I only tested on regular NUMA and not on NUMAQ.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH 00/14] NUMA: Memoryless node support V4

[Lee Schermerhorn]

Changes V3->V4:
- Refresh against 23-rc1-mm1
- teach cpusets about memoryless nodes.

Changes V2->V3:
- Refresh patches (sigh)
- Add comments suggested by Kamezawa Hiroyuki
- Add signoff by Jes Sorensen

Changes V1->V2:
- Add a generic layer that allows the definition of additional node bitmaps

This patchset is implementing additional node bitmaps that allow the system
to track nodes that are online without memory and nodes that have processors.

In various subsystems we can use that information to customize VM behavior.


We define a number of node states that we track in enum node_states

/*
 * Bitmasks that are kept for all the nodes.
 */
enum node_states {
	N_POSSIBLE,     /* The node could become online at some point */
	N_ONLINE,       /* The node is online */
	N_MEMORY,       /* The node has memory */
	N_CPU,          /* The node has cpus */
	NR_NODE_STATES
};

and define operations using the node states:

static inline int node_state(int node, enum node_states state)
{
        return node_isset(node, node_states[state]);
}

static inline void node_set_state(int node, enum node_states state)
{
        __node_set(node, &node_states[state]);
}

static inline void node_clear_state(int node, enum node_states state)
{
        __node_clear(node, &node_states[state]);
}

static inline int num_node_state(enum node_states state)
{
        return nodes_weight(node_states[state]);
}

-- 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH 02/14] Memoryless nodes: introduce mask of nodes with memory

[Lee Schermerhorn]

[patch 2/14] Memoryless nodes:  introduce mask of nodes with memory

It is necessary to know if nodes have memory since we have recently
begun to add support for memoryless nodes. For that purpose we introduce
a new node state N_MEMORY.

A node has its bit in node_memory_map set if it has memory. If a node
has memory then it has at least one zone defined in its pgdat structure
that is located in the pgdat itself.

N_MEMORY can then be used in various places to insure that we
do the right thing when we encounter a memoryless node.

Signed-off-by: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Signed-off-by: Christoph Lameter <clameter@sgi.com>
Tested-off-by: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
Acked-by: Bob Picco <bob.picco@hp.com>

 include/linux/nodemask.h |    1 +
 mm/page_alloc.c          |    9 +++++++--
 2 files changed, 8 insertions(+), 2 deletions(-)

Index: Linux/include/linux/nodemask.h
===================================================================
--- Linux.orig/include/linux/nodemask.h	2007-07-25 11:36:25.000000000 -0400
+++ Linux/include/linux/nodemask.h	2007-07-25 11:36:27.000000000 -0400
@@ -343,6 +343,7 @@ static inline void __nodes_remap(nodemas
 enum node_states {
 	N_POSSIBLE,	/* The node could become online at some point */
 	N_ONLINE,	/* The node is online */
+	N_MEMORY,	/* The node has memory */
 	NR_NODE_STATES
 };
 
Index: Linux/mm/page_alloc.c
===================================================================
--- Linux.orig/mm/page_alloc.c	2007-07-25 11:36:25.000000000 -0400
+++ Linux/mm/page_alloc.c	2007-07-25 11:36:27.000000000 -0400
@@ -2387,8 +2387,13 @@ static int __build_all_zonelists(void *d
 	int nid;
 
 	for_each_online_node(nid) {
-		build_zonelists(NODE_DATA(nid));
-		build_zonelist_cache(NODE_DATA(nid));
+		pg_data_t *pgdat = NODE_DATA(nid);
+
+		build_zonelists(pgdat);
+		build_zonelist_cache(pgdat);
+
+		if (pgdat->node_present_pages)
+			node_set_state(nid, N_MEMORY);
 	}
 	return 0;
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>