[RFC] hotplug, memory: move register_memory_resource out of the lock_memory_hotplug

[RFC] hotplug, memory: move register_memory_resource out of the lock_memory_hotplug

[Nathan Zimmer]

We don't need to do register_memory_resource() since it has its own lock and
doesn't make any callbacks.

Also register_memory_resource return NULL on failure so we don't have anything
to cleanup at this point.


The reason for this rfc is I was doing some experiments with hotplugging of
memory on some of our larger systems.  While it seems to work, it can be quite
slow.  With some preliminary digging I found that lock_memory_hotplug is
clearly ripe for breakup.

It could be broken up per nid or something but it also covers the
online_page_callback.  The online_page_callback shouldn't be very hard to break
out.

Also there is the issue of various structures(wmarks come to mind) that are
only updated under the lock_memory_hotplug that would need to be dealt with.


cc: Andrew Morton <akpm@linux-foundation.org>
cc: Tang Chen <tangchen@cn.fujitsu.com>
cc: Wen Congyang <wency@cn.fujitsu.com>
cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
cc: Hedi <hedi@sgi.com>
cc: Mike Travis <travis@sgi.com>
cc: linux-mm@kvack.org
cc: linux-kernel@vger.kernel.org


---
 mm/memory_hotplug.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 1ad92b4..62a0cd1 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1097,17 +1097,18 @@ int __ref add_memory(int nid, u64 start, u64 size)
 	struct resource *res;
 	int ret;
 
-	lock_memory_hotplug();
-
 	res = register_memory_resource(start, size);
 	ret = -EEXIST;
 	if (!res)
-		goto out;
+		return ret;
 
 	{	/* Stupid hack to suppress address-never-null warning */
 		void *p = NODE_DATA(nid);
 		new_pgdat = !p;
 	}
+
+	lock_memory_hotplug();
+
 	new_node = !node_online(nid);
 	if (new_node) {
 		pgdat = hotadd_new_pgdat(nid, start);
-- 
1.8.2.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC] hotplug, memory: move register_memory_resource out of the lock_memory_hotplug

[Andrew Morton]

On Tue, 14 Jan 2014 12:24:34 -0600 Nathan Zimmer <nzimmer@sgi.com> wrote:

> We don't need to do register_memory_resource() since it has its own lock and
> doesn't make any callbacks.
> 
> Also register_memory_resource return NULL on failure so we don't have anything
> to cleanup at this point.
> 
> 
> The reason for this rfc is I was doing some experiments with hotplugging of
> memory on some of our larger systems.  While it seems to work, it can be quite
> slow.  With some preliminary digging I found that lock_memory_hotplug is
> clearly ripe for breakup.
> 
> It could be broken up per nid or something but it also covers the
> online_page_callback.  The online_page_callback shouldn't be very hard to break
> out.
> 
> Also there is the issue of various structures(wmarks come to mind) that are
> only updated under the lock_memory_hotplug that would need to be dealt with.
>
> ...
>
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1097,17 +1097,18 @@ int __ref add_memory(int nid, u64 start, u64 size)
>  	struct resource *res;
>  	int ret;
>  
> -	lock_memory_hotplug();
> -
>  	res = register_memory_resource(start, size);
>  	ret = -EEXIST;
>  	if (!res)
> -		goto out;
> +		return ret;
>  
>  	{	/* Stupid hack to suppress address-never-null warning */
>  		void *p = NODE_DATA(nid);
>  		new_pgdat = !p;
>  	}
> +
> +	lock_memory_hotplug();
> +
>  	new_node = !node_online(nid);
>  	if (new_node) {
>  		pgdat = hotadd_new_pgdat(nid, start);

Looks sane to me.

register_memory_resource() makes me cry.  Please review:


From: Andrew Morton <akpm@linux-foundation.org>
Subject: mm/memory_hotplug.c: register_memory_resource() fixes

- register_memory_resource() should not go BUG on ENOMEM.  That's
  appropriate at system boot time, but not at memory-hotplug time.  Fix.

- register_memory_resource()'s caller is incorrectly replacing
  request_resource()'s -EBUSY with -EEXIST.  Fix this by propagating
  errors appropriately.

Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Cc: Hedi <hedi@sgi.com>
Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Cc: Mike Travis <travis@sgi.com>
Cc: Nathan Zimmer <nzimmer@sgi.com>
Cc: Tang Chen <tangchen@cn.fujitsu.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 mm/memory_hotplug.c |   15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff -puN mm/memory_hotplug.c~mm-memory_hotplugc-register_memory_resource-fixes mm/memory_hotplug.c
--- a/mm/memory_hotplug.c~mm-memory_hotplugc-register_memory_resource-fixes
+++ a/mm/memory_hotplug.c
@@ -64,17 +64,21 @@ void unlock_memory_hotplug(void)
 static struct resource *register_memory_resource(u64 start, u64 size)
 {
 	struct resource *res;
+	int err;
+
 	res = kzalloc(sizeof(struct resource), GFP_KERNEL);
-	BUG_ON(!res);
+	if (!res)
+		return ERR_PTR(-ENOMEM);
 
 	res->name = "System RAM";
 	res->start = start;
 	res->end = start + size - 1;
 	res->flags = IORESOURCE_MEM | IORESOURCE_BUSY;
-	if (request_resource(&iomem_resource, res) < 0) {
+	err = request_resource(&iomem_resource, res);
+	if (err) {
 		pr_debug("System RAM resource %pR cannot be added\n", res);
 		kfree(res);
-		res = NULL;
+		res = ERR_PTR(err);
 	}
 	return res;
 }
@@ -1108,9 +1112,8 @@ int __ref add_memory(int nid, u64 start,
 		return ret;
 
 	res = register_memory_resource(start, size);
-	ret = -EEXIST;
-	if (!res)
-		return ret;
+	if (IS_ERR(res))
+		return PTR_ERR(res);
 
 	{	/* Stupid hack to suppress address-never-null warning */
 		void *p = NODE_DATA(nid);
_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC] hotplug, memory: move register_memory_resource out of the lock_memory_hotplug

[David Rientjes]

On Tue, 14 Jan 2014, Nathan Zimmer wrote:

> We don't need to do register_memory_resource() since it has its own lock and
> doesn't make any callbacks.
> 

We need to do it, just not under lock_memory_hotplug() :).

> Also register_memory_resource return NULL on failure so we don't have anything
> to cleanup at this point.
> 
> 
> The reason for this rfc is I was doing some experiments with hotplugging of
> memory on some of our larger systems.  While it seems to work, it can be quite
> slow.  With some preliminary digging I found that lock_memory_hotplug is
> clearly ripe for breakup.
> 
> It could be broken up per nid or something but it also covers the
> online_page_callback.  The online_page_callback shouldn't be very hard to break
> out.
> 
> Also there is the issue of various structures(wmarks come to mind) that are
> only updated under the lock_memory_hotplug that would need to be dealt with.
> 
> 
> cc: Andrew Morton <akpm@linux-foundation.org>
> cc: Tang Chen <tangchen@cn.fujitsu.com>
> cc: Wen Congyang <wency@cn.fujitsu.com>
> cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
> cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
> cc: Hedi <hedi@sgi.com>
> cc: Mike Travis <travis@sgi.com>
> cc: linux-mm@kvack.org
> cc: linux-kernel@vger.kernel.org

Looks like you're modifying a pre-3.12 kernel version that doesn't have 
27356f54c8c3 ("mm/hotplug: verify hotplug memory range").

When your patch is signed off, feel free to add

Acked-by: David Rientjes <rientjes@google.com>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC] hotplug, memory: move register_memory_resource out of the lock_memory_hotplug

[David Rientjes]

On Tue, 14 Jan 2014, Andrew Morton wrote:

> From: Andrew Morton <akpm@linux-foundation.org>
> Subject: mm/memory_hotplug.c: register_memory_resource() fixes
> 
> - register_memory_resource() should not go BUG on ENOMEM.  That's
>   appropriate at system boot time, but not at memory-hotplug time.  Fix.
> 
> - register_memory_resource()'s caller is incorrectly replacing
>   request_resource()'s -EBUSY with -EEXIST.  Fix this by propagating
>   errors appropriately.
> 

Unfortunately, -EEXIST is a special case return value for both 
acpi_memory_enable_device() and hv_mem_hot_add(), so they would need to be 
modified to agree concurrently with this change.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC] hotplug, memory: move register_memory_resource out of the lock_memory_hotplug

[Yasuaki Ishimatsu]

(2014/01/15 3:24), Nathan Zimmer wrote:
> We don't need to do register_memory_resource() since it has its own lock and
> doesn't make any callbacks.
> 
> Also register_memory_resource return NULL on failure so we don't have anything
> to cleanup at this point.
> 
> 
> The reason for this rfc is I was doing some experiments with hotplugging of
> memory on some of our larger systems.  While it seems to work, it can be quite
> slow.  With some preliminary digging I found that lock_memory_hotplug is
> clearly ripe for breakup.
> 
> It could be broken up per nid or something but it also covers the
> online_page_callback.  The online_page_callback shouldn't be very hard to break
> out.
> 
> Also there is the issue of various structures(wmarks come to mind) that are
> only updated under the lock_memory_hotplug that would need to be dealt with.
> 
> 
> cc: Andrew Morton <akpm@linux-foundation.org>
> cc: Tang Chen <tangchen@cn.fujitsu.com>
> cc: Wen Congyang <wency@cn.fujitsu.com>
> cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
> cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
> cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
> cc: Hedi <hedi@sgi.com>
> cc: Mike Travis <travis@sgi.com>
> cc: linux-mm@kvack.org
> cc: linux-kernel@vger.kernel.org
> 
> 
> ---

The patch seems good to me.

Reviewed-by: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>

Thanks,
Yasuaki Ishimatsu

>   mm/memory_hotplug.c | 7 ++++---
>   1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 1ad92b4..62a0cd1 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1097,17 +1097,18 @@ int __ref add_memory(int nid, u64 start, u64 size)
>   	struct resource *res;
>   	int ret;
>   
> -	lock_memory_hotplug();
> -
>   	res = register_memory_resource(start, size);
>   	ret = -EEXIST;
>   	if (!res)
> -		goto out;
> +		return ret;
>   
>   	{	/* Stupid hack to suppress address-never-null warning */
>   		void *p = NODE_DATA(nid);
>   		new_pgdat = !p;
>   	}
> +
> +	lock_memory_hotplug();
> +
>   	new_node = !node_online(nid);
>   	if (new_node) {
>   		pgdat = hotadd_new_pgdat(nid, start);
> 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC] hotplug, memory: move register_memory_resource out of the lock_memory_hotplug

[Andrew Morton]

On Tue, 14 Jan 2014 17:05:42 -0800 (PST) David Rientjes <rientjes@google.com> wrote:

> On Tue, 14 Jan 2014, Andrew Morton wrote:
> 
> > From: Andrew Morton <akpm@linux-foundation.org>
> > Subject: mm/memory_hotplug.c: register_memory_resource() fixes
> > 
> > - register_memory_resource() should not go BUG on ENOMEM.  That's
> >   appropriate at system boot time, but not at memory-hotplug time.  Fix.
> > 
> > - register_memory_resource()'s caller is incorrectly replacing
> >   request_resource()'s -EBUSY with -EEXIST.  Fix this by propagating
> >   errors appropriately.
> > 
> 
> Unfortunately, -EEXIST is a special case return value for both 
> acpi_memory_enable_device() and hv_mem_hot_add(), so they would need to be 
> modified to agree concurrently with this change.

blah, OK, thanks, I'll drop it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>