[Patch](memory hotplug) Hot-add with sparsemem-vmemmap

[Patch](memory hotplug) Hot-add with sparsemem-vmemmap

[Yasunori Goto]

Hello.

This patch is to avoid panic when memory hot-add is executed with
sparsemem-vmemmap. Current vmemmap-sparsemem code doesn't support
memory hot-add. Vmemmap must be populated when hot-add.
This is for 2.6.23-rc2-mm2.

Todo: # Even if this patch is applied, the message "[xxxx-xxxx] potential
        offnode page_structs" is displayed. To allocate memmap on its node,
        memmap (and pgdat) must be initialized itself like chicken and
        egg relationship.

      # vmemmap_unpopulate will be necessary for followings.
         - For cancel hot-add due to error.
         - For unplug.

Please comment.

Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>

---
 include/linux/mm.h  |    2 +-
 mm/sparse-vmemmap.c |    2 +-
 mm/sparse.c         |   24 +++++++++++++++++++++---
 3 files changed, 23 insertions(+), 5 deletions(-)

Index: vmemmap/mm/sparse-vmemmap.c
===================================================================
--- vmemmap.orig/mm/sparse-vmemmap.c	2007-08-10 20:17:19.000000000 +0900
+++ vmemmap/mm/sparse-vmemmap.c	2007-08-10 21:12:54.000000000 +0900
@@ -170,7 +170,7 @@ int __meminit vmemmap_populate(struct pa
 }
 #endif /* !CONFIG_ARCH_POPULATES_SPARSEMEM_VMEMMAP */
 
-struct page __init *sparse_early_mem_map_populate(unsigned long pnum, int nid)
+struct page *sparse_mem_map_populate(unsigned long pnum, int nid)
 {
 	struct page *map = pfn_to_page(pnum * PAGES_PER_SECTION);
 	int error = vmemmap_populate(map, PAGES_PER_SECTION, nid);
Index: vmemmap/include/linux/mm.h
===================================================================
--- vmemmap.orig/include/linux/mm.h	2007-08-10 20:17:19.000000000 +0900
+++ vmemmap/include/linux/mm.h	2007-08-10 21:06:34.000000000 +0900
@@ -1146,7 +1146,7 @@ extern int randomize_va_space;
 
 const char * arch_vma_name(struct vm_area_struct *vma);
 
-struct page *sparse_early_mem_map_populate(unsigned long pnum, int nid);
+struct page *sparse_mem_map_populate(unsigned long pnum, int nid);
 int vmemmap_populate(struct page *start_page, unsigned long pages, int node);
 int vmemmap_populate_pmd(pud_t *, unsigned long, unsigned long, int);
 void *vmemmap_alloc_block(unsigned long size, int node);
Index: vmemmap/mm/sparse.c
===================================================================
--- vmemmap.orig/mm/sparse.c	2007-08-10 20:17:19.000000000 +0900
+++ vmemmap/mm/sparse.c	2007-08-10 21:21:01.000000000 +0900
@@ -259,7 +259,7 @@ static unsigned long *sparse_early_usema
 }
 
 #ifndef CONFIG_SPARSEMEM_VMEMMAP
-struct page __init *sparse_early_mem_map_populate(unsigned long pnum, int nid)
+struct page __init *sparse_mem_map_populate(unsigned long pnum, int nid)
 {
 	struct page *map;
 
@@ -284,7 +284,7 @@ struct page __init *sparse_early_mem_map
 	struct mem_section *ms = __nr_to_section(pnum);
 	int nid = sparse_early_nid(ms);
 
-	map = sparse_early_mem_map_populate(pnum, nid);
+	map = sparse_mem_map_populate(pnum, nid);
 	if (map)
 		return map;
 
@@ -322,6 +322,17 @@ void __init sparse_init(void)
 }
 
 #ifdef CONFIG_MEMORY_HOTPLUG
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
+static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid,
+						 unsigned long nr_pages)
+{
+	return sparse_mem_map_populate(pnum, nid);
+}
+static void __kfree_section_memmap(struct page *memmap, unsigned long nr_pages)
+{
+	return; /* XXX: Not implemented yet */
+}
+#else
 static struct page *__kmalloc_section_memmap(unsigned long nr_pages)
 {
 	struct page *page, *ret;
@@ -344,6 +355,12 @@ got_map_ptr:
 	return ret;
 }
 
+static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid,
+						  unsigned long nr_pages)
+{
+	return __kmalloc_section_memmap(nr_pages);
+}
+
 static int vaddr_in_vmalloc_area(void *addr)
 {
 	if (addr >= (void *)VMALLOC_START &&
@@ -360,6 +377,7 @@ static void __kfree_section_memmap(struc
 		free_pages((unsigned long)memmap,
 			   get_order(sizeof(struct page) * nr_pages));
 }
+#endif /* CONFIG_SPARSEMEM_VMEMMAP */
 
 /*
  * returns the number of sections whose mem_maps were properly
@@ -382,7 +400,7 @@ int sparse_add_one_section(struct zone *
 	 * plus, it does a kmalloc
 	 */
 	sparse_index_init(section_nr, pgdat->node_id);
-	memmap = __kmalloc_section_memmap(nr_pages);
+	memmap = kmalloc_section_memmap(section_nr, pgdat->node_id, nr_pages);
 	usemap = __kmalloc_section_usemap();
 
 	pgdat_resize_lock(pgdat, &flags);

-- 
Yasunori Goto 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Patch](memory hotplug) Hot-add with sparsemem-vmemmap

[Christoph Lameter]

On Fri, 17 Aug 2007, Yasunori Goto wrote:

> Todo: # Even if this patch is applied, the message "[xxxx-xxxx] potential
>         offnode page_structs" is displayed. To allocate memmap on its node,
>         memmap (and pgdat) must be initialized itself like chicken and
>         egg relationship.

Hmmmm.... You need to create something like the bootmem allocator?

Or relocate the memory map later.

Or just add a small piece of memory first so that only one memmap block is 
placed off line?
 
>       # vmemmap_unpopulate will be necessary for followings.
>          - For cancel hot-add due to error.
>          - For unplug.
> 
> Please comment.

Looks fine to me.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Patch](memory hotplug) Hot-add with sparsemem-vmemmap

[Yasunori Goto]

> On Fri, 17 Aug 2007, Yasunori Goto wrote:
> 
> > Todo: # Even if this patch is applied, the message "[xxxx-xxxx] potential
> >         offnode page_structs" is displayed. To allocate memmap on its node,
> >         memmap (and pgdat) must be initialized itself like chicken and
> >         egg relationship.
> 
> Hmmmm.... You need to create something like the bootmem allocator?

Right. I suppose it may be better.

> 
> Or relocate the memory map later.

I suppose relocation will be messy way.....

> 
> Or just add a small piece of memory first so that only one memmap block is 
> placed off line?

I'm not sure what you mean about "add small piece".
Do you mean reservation for memmap/pgdat block first?
Then this is second choice.

Even if either is chosen, kernel must remember memmap's place until
unplug. I'll try easier way.

>  
> >       # vmemmap_unpopulate will be necessary for followings.
> >          - For cancel hot-add due to error.
> >          - For unplug.
> > 
> > Please comment.
> 
> Looks fine to me.

Thanks :-)


-- 
Yasunori Goto 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Patch](memory hotplug) Hot-add with sparsemem-vmemmap

[Mel Gorman]

On (17/08/07 16:08), Yasunori Goto didst pronounce:
> Hello.
> 
> This patch is to avoid panic when memory hot-add is executed with
> sparsemem-vmemmap. Current vmemmap-sparsemem code doesn't support
> memory hot-add. Vmemmap must be populated when hot-add.
> This is for 2.6.23-rc2-mm2.
> 
> Todo: # Even if this patch is applied, the message "[xxxx-xxxx] potential
>         offnode page_structs" is displayed. To allocate memmap on its node,
>         memmap (and pgdat) must be initialized itself like chicken and
>         egg relationship.
> 
>       # vmemmap_unpopulate will be necessary for followings.
>          - For cancel hot-add due to error.
>          - For unplug.
> 
> Please comment.
> 
> Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
> 
> ---
>  include/linux/mm.h  |    2 +-
>  mm/sparse-vmemmap.c |    2 +-
>  mm/sparse.c         |   24 +++++++++++++++++++++---
>  3 files changed, 23 insertions(+), 5 deletions(-)
> 
> Index: vmemmap/mm/sparse-vmemmap.c
> ===================================================================
> --- vmemmap.orig/mm/sparse-vmemmap.c	2007-08-10 20:17:19.000000000 +0900
> +++ vmemmap/mm/sparse-vmemmap.c	2007-08-10 21:12:54.000000000 +0900
> @@ -170,7 +170,7 @@ int __meminit vmemmap_populate(struct pa
>  }
>  #endif /* !CONFIG_ARCH_POPULATES_SPARSEMEM_VMEMMAP */
>  
> -struct page __init *sparse_early_mem_map_populate(unsigned long pnum, int nid)
> +struct page *sparse_mem_map_populate(unsigned long pnum, int nid)

__meminit here instead of __init?

>  {
>  	struct page *map = pfn_to_page(pnum * PAGES_PER_SECTION);
>  	int error = vmemmap_populate(map, PAGES_PER_SECTION, nid);
> Index: vmemmap/include/linux/mm.h
> ===================================================================
> --- vmemmap.orig/include/linux/mm.h	2007-08-10 20:17:19.000000000 +0900
> +++ vmemmap/include/linux/mm.h	2007-08-10 21:06:34.000000000 +0900
> @@ -1146,7 +1146,7 @@ extern int randomize_va_space;
>  
>  const char * arch_vma_name(struct vm_area_struct *vma);
>  
> -struct page *sparse_early_mem_map_populate(unsigned long pnum, int nid);
> +struct page *sparse_mem_map_populate(unsigned long pnum, int nid);
>  int vmemmap_populate(struct page *start_page, unsigned long pages, int node);
>  int vmemmap_populate_pmd(pud_t *, unsigned long, unsigned long, int);
>  void *vmemmap_alloc_block(unsigned long size, int node);
> Index: vmemmap/mm/sparse.c
> ===================================================================
> --- vmemmap.orig/mm/sparse.c	2007-08-10 20:17:19.000000000 +0900
> +++ vmemmap/mm/sparse.c	2007-08-10 21:21:01.000000000 +0900
> @@ -259,7 +259,7 @@ static unsigned long *sparse_early_usema
>  }
>  
>  #ifndef CONFIG_SPARSEMEM_VMEMMAP
> -struct page __init *sparse_early_mem_map_populate(unsigned long pnum, int nid)
> +struct page __init *sparse_mem_map_populate(unsigned long pnum, int nid)

__meminit again possibly.

>  {
>  	struct page *map;
>  
> @@ -284,7 +284,7 @@ struct page __init *sparse_early_mem_map
>  	struct mem_section *ms = __nr_to_section(pnum);
>  	int nid = sparse_early_nid(ms);
>  
> -	map = sparse_early_mem_map_populate(pnum, nid);
> +	map = sparse_mem_map_populate(pnum, nid);
>  	if (map)
>  		return map;
>  
> @@ -322,6 +322,17 @@ void __init sparse_init(void)
>  }
>  
>  #ifdef CONFIG_MEMORY_HOTPLUG
> +#ifdef CONFIG_SPARSEMEM_VMEMMAP
> +static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid,
> +						 unsigned long nr_pages)
> +{
> +	return sparse_mem_map_populate(pnum, nid);
> +}

In the other version of __kmalloc_section_memmap(), pages get allocated
from alloc_pages() and it's obvious it's allocated there. A one line
comment saying that sparse_mem_map_populate() will make the necessary
allocations eventually would be nice.

Not a big deal though.

> +static void __kfree_section_memmap(struct page *memmap, unsigned long nr_pages)
> +{
> +	return; /* XXX: Not implemented yet */
> +}
> +#else
>  static struct page *__kmalloc_section_memmap(unsigned long nr_pages)
>  {
>  	struct page *page, *ret;
> @@ -344,6 +355,12 @@ got_map_ptr:
>  	return ret;
>  }
>  
> +static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid,
> +						  unsigned long nr_pages)
> +{
> +	return __kmalloc_section_memmap(nr_pages);
> +}
> +
>  static int vaddr_in_vmalloc_area(void *addr)
>  {
>  	if (addr >= (void *)VMALLOC_START &&
> @@ -360,6 +377,7 @@ static void __kfree_section_memmap(struc
>  		free_pages((unsigned long)memmap,
>  			   get_order(sizeof(struct page) * nr_pages));
>  }
> +#endif /* CONFIG_SPARSEMEM_VMEMMAP */
>  
>  /*
>   * returns the number of sections whose mem_maps were properly
> @@ -382,7 +400,7 @@ int sparse_add_one_section(struct zone *
>  	 * plus, it does a kmalloc
>  	 */
>  	sparse_index_init(section_nr, pgdat->node_id);
> -	memmap = __kmalloc_section_memmap(nr_pages);
> +	memmap = kmalloc_section_memmap(section_nr, pgdat->node_id, nr_pages);
>  	usemap = __kmalloc_section_usemap();
>  
>  	pgdat_resize_lock(pgdat, &flags);
> 

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Patch](memory hotplug) Hot-add with sparsemem-vmemmap

[Yasunori Goto]

> > Index: vmemmap/mm/sparse-vmemmap.c
> > ===================================================================
> > --- vmemmap.orig/mm/sparse-vmemmap.c	2007-08-10 20:17:19.000000000 +0900
> > +++ vmemmap/mm/sparse-vmemmap.c	2007-08-10 21:12:54.000000000 +0900
> > @@ -170,7 +170,7 @@ int __meminit vmemmap_populate(struct pa
> >  }
> >  #endif /* !CONFIG_ARCH_POPULATES_SPARSEMEM_VMEMMAP */
> >  
> > -struct page __init *sparse_early_mem_map_populate(unsigned long pnum, int nid)
> > +struct page *sparse_mem_map_populate(unsigned long pnum, int nid)
> 
> __meminit here instead of __init?

Ah, Yes. Thanks. I'll fix it.

>
> > Index: vmemmap/mm/sparse.c
> > ===================================================================
> > --- vmemmap.orig/mm/sparse.c	2007-08-10 20:17:19.000000000 +0900
> > +++ vmemmap/mm/sparse.c	2007-08-10 21:21:01.000000000 +0900
> > @@ -259,7 +259,7 @@ static unsigned long *sparse_early_usema
> >  }
> >  
> >  #ifndef CONFIG_SPARSEMEM_VMEMMAP
> > -struct page __init *sparse_early_mem_map_populate(unsigned long pnum, int nid)
> > +struct page __init *sparse_mem_map_populate(unsigned long pnum, int nid)
> 
> __meminit again possibly.

Here should use __init. It is called at boot time and uses
alloc_bootmem(). 


> >  #ifdef CONFIG_MEMORY_HOTPLUG
> > +#ifdef CONFIG_SPARSEMEM_VMEMMAP
> > +static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid,
> > +						 unsigned long nr_pages)
> > +{
> > +	return sparse_mem_map_populate(pnum, nid);
> > +}
> 
> In the other version of __kmalloc_section_memmap(), pages get allocated
> from alloc_pages() and it's obvious it's allocated there. A one line
> comment saying that sparse_mem_map_populate() will make the necessary
> allocations eventually would be nice.
> 
> Not a big deal though.

Ah, Ok. I'll add its comment.

Thanks.

-- 
Yasunori Goto 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[Patch](memory hotplug) Tiny update for hot-add with sparsemem-vmemmap

[Yasunori Goto]

This is tiny update for Mel-san's comment about 
memory hotplug with sparse-vmemmap.

  - Add __meminit to sparse_mem_map_populate()
  - Add a comment.

This is for 2.6.23-rc3-mm1.

Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>

---
 mm/sparse-vmemmap.c |    2 +-
 mm/sparse.c         |    1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

Index: current/mm/sparse-vmemmap.c
===================================================================
--- current.orig/mm/sparse-vmemmap.c	2007-08-23 16:19:10.000000000 +0900
+++ current/mm/sparse-vmemmap.c	2007-08-30 19:25:16.000000000 +0900
@@ -137,7 +137,7 @@ int __meminit vmemmap_populate_basepages
 	return 0;
 }
 
-struct page *sparse_mem_map_populate(unsigned long pnum, int nid)
+struct page * __meminit sparse_mem_map_populate(unsigned long pnum, int nid)
 {
 	struct page *map = pfn_to_page(pnum * PAGES_PER_SECTION);
 	int error = vmemmap_populate(map, PAGES_PER_SECTION, nid);
Index: current/mm/sparse.c
===================================================================
--- current.orig/mm/sparse.c	2007-08-23 16:19:10.000000000 +0900
+++ current/mm/sparse.c	2007-08-30 19:31:50.000000000 +0900
@@ -326,6 +326,7 @@ void __init sparse_init(void)
 static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid,
 						 unsigned long nr_pages)
 {
+	/* This will make the necessary allocations eventually. */
 	return sparse_mem_map_populate(pnum, nid);
 }
 static void __kfree_section_memmap(struct page *memmap, unsigned long nr_pages)

-- 
Yasunori Goto 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Patch](memory hotplug) Tiny update for hot-add with sparsemem-vmemmap

[Mel Gorman]

On (30/08/07 20:04), Yasunori Goto didst pronounce:
> This is tiny update for Mel-san's comment about 
> memory hotplug with sparse-vmemmap.
> 
>   - Add __meminit to sparse_mem_map_populate()
>   - Add a comment.
> 
> This is for 2.6.23-rc3-mm1.
> 
> Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
> 
> ---
>  mm/sparse-vmemmap.c |    2 +-
>  mm/sparse.c         |    1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
> 
> Index: current/mm/sparse-vmemmap.c
> ===================================================================
> --- current.orig/mm/sparse-vmemmap.c	2007-08-23 16:19:10.000000000 +0900
> +++ current/mm/sparse-vmemmap.c	2007-08-30 19:25:16.000000000 +0900
> @@ -137,7 +137,7 @@ int __meminit vmemmap_populate_basepages
>  	return 0;
>  }
>  
> -struct page *sparse_mem_map_populate(unsigned long pnum, int nid)
> +struct page * __meminit sparse_mem_map_populate(unsigned long pnum, int nid)

Looks good

>  {
>  	struct page *map = pfn_to_page(pnum * PAGES_PER_SECTION);
>  	int error = vmemmap_populate(map, PAGES_PER_SECTION, nid);
> Index: current/mm/sparse.c
> ===================================================================
> --- current.orig/mm/sparse.c	2007-08-23 16:19:10.000000000 +0900
> +++ current/mm/sparse.c	2007-08-30 19:31:50.000000000 +0900
> @@ -326,6 +326,7 @@ void __init sparse_init(void)
>  static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid,
>  						 unsigned long nr_pages)
>  {
> +	/* This will make the necessary allocations eventually. */
>  	return sparse_mem_map_populate(pnum, nid);

Not the greatest comment but sufficient to tell the reader that
allocations happen ultimately and that is enough to avoid confusion.

>  }
>  static void __kfree_section_memmap(struct page *memmap, unsigned long nr_pages)
> 
> -- 
> Yasunori Goto 
> 

Thanks a lot.

Acked-by: Mel Gorman <mel@csn.ul.ie>

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>