Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

[Andrew Morton]

On Thu, 23 Aug 2007 10:22:26 -0700
"Luck, Tony" <tony.luck@intel.com> wrote:

> > __get_free_pages() of swiotlb_alloc_coherent() fails in rc3-mm1.
> > But, it doesn't fail on rc2-mm2, and kernel can boot up.
> 
> That looks to be part of the problem here ... failing an order=3
> allocation during boot on a system that just a few lines earlier
> in the boot log reported "Memory: 37474000k/37680640k available"
> looks bad ... but perhaps having *more* memory is part of your problem.
> You may have run low on GFP_DMA memory because some allocation
> scaled by memory size has chewed up a lot of your memory.  To check
> this try booting with a "mem=4G" parameter and see if that helps
> you.
> 
> But it is also bad that the swiotlb() code failed to handle this.
> Can you check whether the problem is related to the size of the
> allocation being just over 256K (a magic number for swiotlb since
> IO_TLB_SEGSIZE is 128 times a slab size of 2k).  Try changing
> lib/swiotlb.c to set IO_TLB_SEGSIZE to 256 instead.
> 

Others are reporting machines which fail int he memory allcoator much
earlier, and which claim to have four CPUs and 16 nodes.  So something is
very wonky in the rc3-mm1 page allocator.

I guess suspicion has to be directed at the memoryless-nodes patches, but
until that's cleared up I don't think there's much to be gained from
chasing this iommu problem, now that you've worked out that it's a bogus
memory allocation failure (thanks).


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

[Yasunori Goto]

I found find_next_best_node() was wrong.
I confirmed boot up by the following patch.
Mel-san, Kamalesh-san, could you try this?

Bye.
---

Fix decision of memoryless node in find_next_best_node().
This can be cause of SW-IOMMU's allocation failure.

This patch is for 2.6.23-rc3-mm1.

Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>

---
 mm/page_alloc.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: current/mm/page_alloc.c
===================================================================
--- current.orig/mm/page_alloc.c	2007-08-24 16:03:17.000000000 +0900
+++ current/mm/page_alloc.c	2007-08-24 16:04:06.000000000 +0900
@@ -2136,7 +2136,7 @@ static int find_next_best_node(int node,
 		 * Note:  N_HIGH_MEMORY state not guaranteed to be
 		 *        populated yet.
 		 */
-		if (pgdat->node_present_pages)
+		if (!pgdat->node_present_pages)
 			continue;
 
 		/* Don't want a node to appear more than once */

-- 
Yasunori Goto 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

[Mel Gorman]

On (24/08/07 15:53), Yasunori Goto didst pronounce:
> 
> I found find_next_best_node() was wrong.
> I confirmed boot up by the following patch.
> Mel-san, Kamalesh-san, could you try this?
> 

This boots the IA-64 successful and gets rid of that DMA corrupts
memory message. As a bonus, it fixes up the memoryless nodes (the bug
where Total pages == 0 and there is a BUG in page_alloc.c) by building
zonelists properly. The machine still fails to boot with the more familiar
net/core/skbuff.c:95 but that is a separate problem.

Well spotted Yasunori-san.

Andrew, this fixes a real problem and should be considered a fix to
memoryless-nodes-fixup-uses-of-node_online_map-in-generic-code.patch unless
Christoph Lameter objects.

> Bye.
> ---
> 
> Fix decision of memoryless node in find_next_best_node().
> This can be cause of SW-IOMMU's allocation failure.
> 
> This patch is for 2.6.23-rc3-mm1.
> 
> Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
> 

Acked-by: Mel Gorman <mel@csn.ul.ie>

> ---
>  mm/page_alloc.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> Index: current/mm/page_alloc.c
> ===================================================================
> --- current.orig/mm/page_alloc.c	2007-08-24 16:03:17.000000000 +0900
> +++ current/mm/page_alloc.c	2007-08-24 16:04:06.000000000 +0900
> @@ -2136,7 +2136,7 @@ static int find_next_best_node(int node,
>  		 * Note:  N_HIGH_MEMORY state not guaranteed to be
>  		 *        populated yet.
>  		 */
> -		if (pgdat->node_present_pages)
> +		if (!pgdat->node_present_pages)
>  			continue;
>  
>  		/* Don't want a node to appear more than once */
> 

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

[Lee Schermerhorn]

On Fri, 2007-08-24 at 15:52 +0100, Mel Gorman wrote:
> On (24/08/07 15:53), Yasunori Goto didst pronounce:
> > 
> > I found find_next_best_node() was wrong.
> > I confirmed boot up by the following patch.
> > Mel-san, Kamalesh-san, could you try this?
> > 
> 
> This boots the IA-64 successful and gets rid of that DMA corrupts
> memory message. As a bonus, it fixes up the memoryless nodes (the bug
> where Total pages == 0 and there is a BUG in page_alloc.c) by building
> zonelists properly. The machine still fails to boot with the more familiar
> net/core/skbuff.c:95 but that is a separate problem.
> 
> Well spotted Yasunori-san.
> 
> Andrew, this fixes a real problem and should be considered a fix to
> memoryless-nodes-fixup-uses-of-node_online_map-in-generic-code.patch unless
> Christoph Lameter objects.

I reworked that patch and posted the update on 16aug which does not have
this problem:

http://marc.info/?l=linux-mm&m=118729871101418&w=4

This should replace
memoryless-nodes-fixup-uses-of-node_online_map-in-generic-code.patch
in -mm.

Lee


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

[Kamalesh Babulal]

Yasunori Goto wrote:
> I found find_next_best_node() was wrong.
> I confirmed boot up by the following patch.
> Mel-san, Kamalesh-san, could you try this?
>
> Bye.
> ---
>
> Fix decision of memoryless node in find_next_best_node().
> This can be cause of SW-IOMMU's allocation failure.
>
> This patch is for 2.6.23-rc3-mm1.
>
> Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
>
> ---
>  mm/page_alloc.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> Index: current/mm/page_alloc.c
> ===================================================================
> --- current.orig/mm/page_alloc.c	2007-08-24 16:03:17.000000000 +0900
> +++ current/mm/page_alloc.c	2007-08-24 16:04:06.000000000 +0900
> @@ -2136,7 +2136,7 @@ static int find_next_best_node(int node,
>  		 * Note:  N_HIGH_MEMORY state not guaranteed to be
>  		 *        populated yet.
>  		 */
> -		if (pgdat->node_present_pages)
> +		if (!pgdat->node_present_pages)
>  			continue;
>
>  		/* Don't want a node to appear more than once */
>
>   
This patch resolves the kernel panic problem.

-
Kamalesh Babulal.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

[Christoph Lameter]

On Fri, 24 Aug 2007, Lee Schermerhorn wrote:

> I reworked that patch and posted the update on 16aug which does not have
> this problem:
> 
> http://marc.info/?l=linux-mm&m=118729871101418&w=4
> 
> This should replace
> memoryless-nodes-fixup-uses-of-node_online_map-in-generic-code.patch
> in -mm.

Could you post a diff to rc3-mm1 of that patch?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

[Christoph Lameter]

On Fri, 24 Aug 2007, Mel Gorman wrote:

> Andrew, this fixes a real problem and should be considered a fix to
> memoryless-nodes-fixup-uses-of-node_online_map-in-generic-code.patch unless
> Christoph Lameter objects.

Right. Lets make sure to cc Lee on future discussions of the memoryless 
node patchset.

Acked-by: Christoph Lameter <clameter@sgi.com>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

[Lee Schermerhorn]

On Fri, 2007-08-24 at 10:00 -0700, Christoph Lameter wrote:
> On Fri, 24 Aug 2007, Lee Schermerhorn wrote:
> 
> > I reworked that patch and posted the update on 16aug which does not have
> > this problem:
> > 
> > http://marc.info/?l=linux-mm&m=118729871101418&w=4
> > 
> > This should replace
> > memoryless-nodes-fixup-uses-of-node_online_map-in-generic-code.patch
> > in -mm.
> 
> Could you post a diff to rc3-mm1 of that patch?

Sure.  Here it is.  This looks nicer to me than explicitly skipping
unpopulated nodes in find_next_best_node()--as I tried to do, but
botched it :-(.  I didn't notice that because I'd moved on to v2 before
testing with any significant load.  Even when I was running with v1 with
botched zonelists, I apparently had sufficient memory on each node that
I never had to fallback.

I also didn't notice that Andrew had added v1 instead of v2 to the mm
tree.  Will pay more attention in the future, I promise.

Lee
---------------------------
PATCH Diffs between "Fix generic usage of node_online_map" V1 & V2

Against 2.6.23-rc3-mm1

V1 -> V2:
+ moved population of N_HIGH_MEMORY node state mask to
  free_area_init_nodes(), as this is called before we
  build zonelists.  So, we can use this mask in 
  find_next_best_node.  Still need to keep the duplicate
  code in early_calculate_totalpages() for zone movable
  setup.

mm/page_alloc.c:find_next_best_node()

	visit only nodes with memory [N_HIGH_MEMORY mask]
	looking for next best node for fallback zonelists.

mm/page_alloc.c:find_zone_movable_pfns_for_nodes()

	spread kernelcore over nodes with memory.

	This required calling early_calculate_totalpages()
	unconditionally, and populating N_HIGH_MEMORY node
	state therein from nodes in the early_node_map[].
	This duplicates the code in free_area_init_nodes(), but
	I don't want to depend on this copy if ZONE_MOVABLE 
	might go away, taking early_calculate_totalpages()
	with it.

Signed-off-by:  Lee Schermerhorn <lee.schermerhorn@hp.com>

 mm/page_alloc.c |   48 ++++++++++++++++++++----------------------------
 1 file changed, 20 insertions(+), 28 deletions(-)

Index: Linux/mm/page_alloc.c
===================================================================
--- Linux.orig/mm/page_alloc.c	2007-08-24 13:20:28.000000000 -0400
+++ Linux/mm/page_alloc.c	2007-08-24 13:25:20.000000000 -0400
@@ -2127,18 +2127,10 @@ static int find_next_best_node(int node,
 		return node;
 	}
 
-	for_each_online_node(n) {
+	for_each_node_state(n, N_HIGH_MEMORY) {
 		pg_data_t *pgdat = NODE_DATA(n);
 		cpumask_t tmp;
 
-		/*
-		 * skip nodes w/o memory.
-		 * Note:  N_HIGH_MEMORY state not guaranteed to be
-		 *        populated yet.
-		 */
-		if (pgdat->node_present_pages)
-			continue;
-
 		/* Don't want a node to appear more than once */
 		if (node_isset(n, *used_node_mask))
 			continue;
@@ -2433,20 +2425,6 @@ static void build_zonelist_cache(pg_data
 
 #endif	/* CONFIG_NUMA */
 
-/* Any regular memory on that node ? */
-static void check_for_regular_memory(pg_data_t *pgdat)
-{
-#ifdef CONFIG_HIGHMEM
-	enum zone_type zone_type;
-
-	for (zone_type = 0; zone_type <= ZONE_NORMAL; zone_type++) {
-		struct zone *zone = &pgdat->node_zones[zone_type];
-		if (zone->present_pages)
-			node_set_state(zone_to_nid(zone), N_NORMAL_MEMORY);
-	}
-#endif
-}
-
 /* return values int ....just for stop_machine_run() */
 static int __build_all_zonelists(void *dummy)
 {
@@ -2457,11 +2435,6 @@ static int __build_all_zonelists(void *d
 
 		build_zonelists(pgdat);
 		build_zonelist_cache(pgdat);
-
-		/* Any memory on that node */
-		if (pgdat->node_present_pages)
-			node_set_state(nid, N_HIGH_MEMORY);
-		check_for_regular_memory(pgdat);
 	}
 	return 0;
 }
@@ -3919,6 +3892,20 @@ restart:
 			roundup(zone_movable_pfn[nid], MAX_ORDER_NR_PAGES);
 }
 
+/* Any regular memory on that node ? */
+static void check_for_regular_memory(pg_data_t *pgdat)
+{
+#ifdef CONFIG_HIGHMEM
+	enum zone_type zone_type;
+
+	for (zone_type = 0; zone_type <= ZONE_NORMAL; zone_type++) {
+		struct zone *zone = &pgdat->node_zones[zone_type];
+		if (zone->present_pages)
+			node_set_state(zone_to_nid(zone), N_NORMAL_MEMORY);
+	}
+#endif
+}
+
 /**
  * free_area_init_nodes - Initialise all pg_data_t and zone data
  * @max_zone_pfn: an array of max PFNs for each zone
@@ -3996,6 +3983,11 @@ void __init free_area_init_nodes(unsigne
 		pg_data_t *pgdat = NODE_DATA(nid);
 		free_area_init_node(nid, pgdat, NULL,
 				find_min_pfn_for_node(nid), NULL);
+
+		/* Any memory on that node */
+		if (pgdat->node_present_pages)
+			node_set_state(nid, N_HIGH_MEMORY);
+		check_for_regular_memory(pgdat);
 	}
 }
 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

[Christoph Lameter]

On Fri, 24 Aug 2007, Lee Schermerhorn wrote:

> PATCH Diffs between "Fix generic usage of node_online_map" V1 & V2

Ahh. Yes. I remember some of that.

Acked-by: Christoph Lameter <clameter@sgi.com>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

[Adam Litke]

On Fri, 2007-08-24 at 15:53 +0900, Yasunori Goto wrote:
> I found find_next_best_node() was wrong.
> I confirmed boot up by the following patch.
> Mel-san, Kamalesh-san, could you try this?

FYI: This patch also allows the alloc-instantiate-race testcase in
libhugetlbfs to pass again :)

-- 
Adam Litke - (agl at us.ibm.com)
IBM Linux Technology Center

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>