Hotplug memory remove

Hotplug memory remove

[Badari Pulavarty]

Hi,

I am trying to test hotplug memory remove support on ppc64.
I have few questions (sorry if they are stupid), hoping you could 
help me.

1) Other than remove_memory(), I don't see any other arch-specific
code that needs to be provided. Even remove_memory() looks pretty
arch independent. Isn't it ?

2) I copied remove_memory() from IA64 to PPC64. When I am testing
hotplug-remove (echo offline > state), I am not able to remove
any memory at all. I get different type of failures like ..

memory offlining 6e000 to 6f000 failed

- OR -

Offlined Pages 0

I am wondering, how did you test it on IA64 ? Am I missing something ?
How can I find which "sections" of the memory are free to remove ?
I am using /proc/page_owner to figure it out for now.


Thanks,
Badari

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[KAMEZAWA Hiroyuki]

On Mon, 01 Oct 2007 08:37:43 -0700
Badari Pulavarty <pbadari@gmail.com> wrote:
> 1) Other than remove_memory(), I don't see any other arch-specific
> code that needs to be provided. Even remove_memory() looks pretty
> arch independent. Isn't it ?
> 
Yes, maybe arch independent. Current codes is based on assumption
that some arch may needs some code before/after hotremove.
If no arch needs, we can merge all. 

> 2) I copied remove_memory() from IA64 to PPC64. When I am testing
> hotplug-remove (echo offline > state), I am not able to remove
> any memory at all. I get different type of failures like ..
> 
> memory offlining 6e000 to 6f000 failed
> 
I'm not sure about this...does this memory is in ZONE_MOVABLE ?
If not ZONE_MOVABLE, offlining can be fail because of not-removable
kernel memory. 

> - OR -
> 
> Offlined Pages 0
> 
Hmm, About "Offlined Pages 0" case, maybe memory resource is not
registered. At memory hotremove works based on registered memory resource.
(For handling memory hole.)

Does PPC64 resister conventinal memory to memory resource ?
This information can be shown in /proc/iomem.
In current code, removable memory must be registerred in /proc/iomem.
Could you confirm ?

> I am wondering, how did you test it on IA64 ? Am I missing something ?
> How can I find which "sections" of the memory are free to remove ?
> I am using /proc/page_owner to figure it out for now.
> 
create ZONE_MOVBALE with kernelcore= boot option and offlined memory in it.
ia64 registers all available memory information to /proc/iomem.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[Badari Pulavarty]

On Tue, 2007-10-02 at 01:14 +0900, KAMEZAWA Hiroyuki wrote:
> On Mon, 01 Oct 2007 08:37:43 -0700
> Badari Pulavarty <pbadari@gmail.com> wrote:
> > 1) Other than remove_memory(), I don't see any other arch-specific
> > code that needs to be provided. Even remove_memory() looks pretty
> > arch independent. Isn't it ?
> > 
> Yes, maybe arch independent. Current codes is based on assumption
> that some arch may needs some code before/after hotremove.
> If no arch needs, we can merge all. 

Yeah. Lets not worry about it yet. All I wanted to make sure,
if there is any arch specific work you did so far..

> 
> > 2) I copied remove_memory() from IA64 to PPC64. When I am testing
> > hotplug-remove (echo offline > state), I am not able to remove
> > any memory at all. I get different type of failures like ..
> > 
> > memory offlining 6e000 to 6f000 failed
> > 
> I'm not sure about this...does this memory is in ZONE_MOVABLE ?
> If not ZONE_MOVABLE, offlining can be fail because of not-removable
> kernel memory. 

I tried offlining different sections of memory. There is no easy 
way to tell if it belonged to ZONE_MOVABLE or not. I was
using /proc/page_owner to find out suitable sections to offline.

> 
> > - OR -
> > 
> > Offlined Pages 0
> > 
> Hmm, About "Offlined Pages 0" case, maybe memory resource is not
> registered. At memory hotremove works based on registered memory resource.
> (For handling memory hole.)
> 
> Does PPC64 resister conventinal memory to memory resource ?
> This information can be shown in /proc/iomem.
> In current code, removable memory must be registerred in /proc/iomem.
> Could you confirm ?

I am little confused. Can you point me to the code where you have
this assumption ? Why does it have to be registered in /proc/meminfo ?
You find the section and try to offline it by migrating pages from that
section. If its fails to free up the pages, fail the remove. Isn't it ?

On my ppc64 machine, I don't see nothing but iomemory in /proc/meminfo.

> > I am wondering, how did you test it on IA64 ? Am I missing something ?
> > How can I find which "sections" of the memory are free to remove ?
> > I am using /proc/page_owner to figure it out for now.
> > 
> create ZONE_MOVBALE with kernelcore= boot option and offlined memory in it.

Will try that.

Thanks,
Badari


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[KAMEZAWA Hiroyuki]

On Mon, 01 Oct 2007 10:49:46 -0700
Badari Pulavarty <pbadari@gmail.com> wrote:

> > 
> > > 2) I copied remove_memory() from IA64 to PPC64. When I am testing
> > > hotplug-remove (echo offline > state), I am not able to remove
> > > any memory at all. I get different type of failures like ..
> > > 
> > > memory offlining 6e000 to 6f000 failed
> > > 
> > I'm not sure about this...does this memory is in ZONE_MOVABLE ?
> > If not ZONE_MOVABLE, offlining can be fail because of not-removable
> > kernel memory. 
> 
> I tried offlining different sections of memory. There is no easy 
> way to tell if it belonged to ZONE_MOVABLE or not. I was
> using /proc/page_owner to find out suitable sections to offline.
> 
Hmm, I myself cat /proc/zoneinfo to know where is ZONE_MOVABLE.
[start_pfn, start_pfn+spanned) is zone range.
But I hear PPC? can overlap plural zones' range...

Some interface like
/sys/device/system/memory/memoryXXX/zone_id
is maybe good, but a section can belongs to multiple zones.


> > Does PPC64 resister conventinal memory to memory resource ?
> > This information can be shown in /proc/iomem.
> > In current code, removable memory must be registerred in /proc/iomem.
> > Could you confirm ?
> 
> I am little confused. Can you point me to the code where you have
> this assumption ? Why does it have to be registered in /proc/meminfo ?
> You find the section and try to offline it by migrating pages from that
> section. If its fails to free up the pages, fail the remove. Isn't it ?
> 
Maybe already you noticed, walk_memory_resource() handles it.

Because a section can includes memory hole or memory for I/O,
we cannot know whether Pg_reserved memory is just reserved or memory hole
or for I/O.

/proc/iomem shows the range of conventional memory if configured in sane way.
For avoiding memory hole, /proc/iomem  gives us very clear resource range
information.

> On my ppc64 machine, I don't see nothing but iomemory in /proc/meminfo.
> 
Ah, not meminfo.
This is my box's proc iomem
==
[kamezawa@drpq linux-2.6.23-rc8-mm2]$ cat /proc/iomem | grep RAM
00000000-0009ffff : System RAM
00100000-03ffffff : System RAM
04000000-04dabfff : System RAM
04dac000-6b4bffff : System RAM
6b4c0000-6b797fff : System RAM
6b798000-6b799fff : System RAM
6b79a000-6b79dfff : System RAM
6b79e000-6b79efff : System RAM
6b79f000-6b7fbfff : System RAM
6b7fc000-6c629fff : System RAM
6c62a000-6c800fff : System RAM
6c801000-6c843fff : System RAM
6c844000-6c847fff : System RAM
6c848000-6c849fff : System RAM
6c84a000-6c85dfff : System RAM
6c85e000-6c85efff : System RAM
6c85f000-6cbfbfff : System RAM
6cbfc000-6d349fff : System RAM
6d34a000-6d3fbfff : System RAM
6d3fc000-6d455fff : System RAM
6d4fc000-6d773fff : System RAM
100000000-7ffffffff : System RAM
4080000000-40ffffffff : System RAM
14004000000-147ffffffff : System RAM
==
All conventional memory is registered as System RAM. This is done in
efi(firmware)'s memory map parsing in ia64.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[Badari Pulavarty]

On Tue, 2007-10-02 at 09:52 +0900, KAMEZAWA Hiroyuki wrote:
> On Mon, 01 Oct 2007 10:49:46 -0700
> Badari Pulavarty <pbadari@gmail.com> wrote:
> 
> > > 
> > > > 2) I copied remove_memory() from IA64 to PPC64. When I am testing
> > > > hotplug-remove (echo offline > state), I am not able to remove
> > > > any memory at all. I get different type of failures like ..
> > > > 
> > > > memory offlining 6e000 to 6f000 failed
> > > > 
> > > I'm not sure about this...does this memory is in ZONE_MOVABLE ?
> > > If not ZONE_MOVABLE, offlining can be fail because of not-removable
> > > kernel memory. 
> > 
> > I tried offlining different sections of memory. There is no easy 
> > way to tell if it belonged to ZONE_MOVABLE or not. I was
> > using /proc/page_owner to find out suitable sections to offline.
> > 
> Hmm, I myself cat /proc/zoneinfo to know where is ZONE_MOVABLE.
> [start_pfn, start_pfn+spanned) is zone range.
> But I hear PPC? can overlap plural zones' range...
> 
> Some interface like
> /sys/device/system/memory/memoryXXX/zone_id
> is maybe good, but a section can belongs to multiple zones.
> 
> 
> > > Does PPC64 resister conventinal memory to memory resource ?
> > > This information can be shown in /proc/iomem.
> > > In current code, removable memory must be registerred in /proc/iomem.
> > > Could you confirm ?
> > 
> > I am little confused. Can you point me to the code where you have
> > this assumption ? Why does it have to be registered in /proc/meminfo ?
> > You find the section and try to offline it by migrating pages from that
> > section. If its fails to free up the pages, fail the remove. Isn't it ?
> > 
> Maybe already you noticed, walk_memory_resource() handles it.
> 
> Because a section can includes memory hole or memory for I/O,
> we cannot know whether Pg_reserved memory is just reserved or memory hole
> or for I/O.
> 
> /proc/iomem shows the range of conventional memory if configured in sane way.
> For avoiding memory hole, /proc/iomem  gives us very clear resource range
> information.
> 
> > On my ppc64 machine, I don't see nothing but iomemory in /proc/meminfo.
> > 
> Ah, not meminfo.
> This is my box's proc iomem

I meant /proc/iomem. On PPC64 all the memory information is in "lmb"
structures and not available through /proc/iomem. We need to find a
way to get to that. I will ask around on ppc mailing list to find
out if there is another easy way.

Thanks,
Badari

elm3b155:~/linux-2.6.23-rc8 # cat /proc/iomem
40080000000-400bfffffff : /pci@800000020000002
  40080000000-400a7ffffff : PCI Bus 0000:d0
    40080000000-400807fffff : 0000:d0:01.0
      40080000000-400807fffff : ipr
    40080800000-400808fffff : 0000:d0:01.0
    40080900000-4008093ffff : 0000:d0:01.0
      40080900000-4008093ffff : ipr
  400a8000000-400afffffff : PCI Bus 0000:cc
    400a8000000-400a8003fff : 0000:cc:01.0
  400b0000000-400b7ffffff : PCI Bus 0000:c8
    400b0000000-400b0000fff : 0000:c8:01.1
    400b0001000-400b0001fff : 0000:c8:01.0
    400b0002000-400b00020ff : 0000:c8:01.2
  400b8000000-400bfefffff : PCI Bus 0000:c0
    400b8000000-400b803ffff : 0000:c0:01.1
    400b8040000-400b807ffff : 0000:c0:01.1
      400b8040000-400b807ffff : e1000
    400b8080000-400b80bffff : 0000:c0:01.0
    400b80c0000-400b80fffff : 0000:c0:01.0
      400b80c0000-400b80fffff : e1000
    400b8100000-400b811ffff : 0000:c0:01.1
      400b8100000-400b811ffff : e1000
    400b8120000-400b813ffff : 0000:c0:01.0
      400b8120000-400b813ffff : e1000
401c0000000-401ffffffff : /pci@800000020000003
  401c0000000-401efffffff : PCI Bus 0001:d8
  401f0000000-401f7ffffff : PCI Bus 0001:c8
  401f8000000-401ffefffff : PCI Bus 0001:d0


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[Badari Pulavarty]

On Tue, 2007-10-02 at 09:52 +0900, KAMEZAWA Hiroyuki wrote:
> On Mon, 01 Oct 2007 10:49:46 -0700
> Badari Pulavarty <pbadari@gmail.com> wrote:
> 
> > > 
> > > > 2) I copied remove_memory() from IA64 to PPC64. When I am testing
> > > > hotplug-remove (echo offline > state), I am not able to remove
> > > > any memory at all. I get different type of failures like ..
> > > > 
> > > > memory offlining 6e000 to 6f000 failed
> > > > 
> > > I'm not sure about this...does this memory is in ZONE_MOVABLE ?
> > > If not ZONE_MOVABLE, offlining can be fail because of not-removable
> > > kernel memory. 
> > 
> > I tried offlining different sections of memory. There is no easy 
> > way to tell if it belonged to ZONE_MOVABLE or not. I was
> > using /proc/page_owner to find out suitable sections to offline.
> > 
> Hmm, I myself cat /proc/zoneinfo to know where is ZONE_MOVABLE.
> [start_pfn, start_pfn+spanned) is zone range.
> But I hear PPC? can overlap plural zones' range...
> 
> Some interface like
> /sys/device/system/memory/memoryXXX/zone_id
> is maybe good, but a section can belongs to multiple zones.
> 
> 
> > > Does PPC64 resister conventinal memory to memory resource ?
> > > This information can be shown in /proc/iomem.
> > > In current code, removable memory must be registerred in /proc/iomem.
> > > Could you confirm ?
> > 
> > I am little confused. Can you point me to the code where you have
> > this assumption ? Why does it have to be registered in /proc/meminfo ?
> > You find the section and try to offline it by migrating pages from that
> > section. If its fails to free up the pages, fail the remove. Isn't it ?
> > 
> Maybe already you noticed, walk_memory_resource() handles it.
> 
> Because a section can includes memory hole or memory for I/O,
> we cannot know whether Pg_reserved memory is just reserved or memory hole
> or for I/O.
> 
> /proc/iomem shows the range of conventional memory if configured in sane way.
> For avoiding memory hole, /proc/iomem  gives us very clear resource range
> information.

Kame,

With little bit of hacking /proc/iomem on ppc64, I got hotplug memory
remove working. I didn't have to spend lot of time debugging the
infrastructure you added. Good work !!

Only complaint I have is, the use of /proc/iomem for verification.
I see few issues.

1) On X86-64, /proc/iomem contains all the memory regions, but they
are all marked IORESOURCE_BUSY. So looking for IORESOURCE_MEM wouldn't
work and always fails. Is any one working on x86-64 ? 

2) On ppc64, /proc/iomem shows only io-mapped-memory regions. So I
had to hack it to add all the memory information. I am going to ask
on ppc64 mailing list on how to do it sanely, but I am afraid that
they are going to say "all the information is available in the kernel
data (lmb) structures, parse them - rather than exporting it
to /proc/iomem". 

We may have to have arch-specific hooks to verify a memory region :(
What do you think ?



Thanks,
Badari

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[KAMEZAWA Hiroyuki]

On Tue, 02 Oct 2007 10:17:34 -0700
Badari Pulavarty <pbadari@gmail.com> wrote:

> Kame,
> 
> With little bit of hacking /proc/iomem on ppc64, I got hotplug memory
> remove working. I didn't have to spend lot of time debugging the
> infrastructure you added. Good work !!
> 
I'm very glad to hear that. Thanks!

> Only complaint I have is, the use of /proc/iomem for verification.
> I see few issues.
> 
> 1) On X86-64, /proc/iomem contains all the memory regions, but they
> are all marked IORESOURCE_BUSY. So looking for IORESOURCE_MEM wouldn't
> work and always fails. Is any one working on x86-64 ? 
> 
no one works on x86-64. But I should ask to x86-64 peaple "Why IORESOURCE_BUSY?"
Thank you for pointing out.

> 2) On ppc64, /proc/iomem shows only io-mapped-memory regions. So I
> had to hack it to add all the memory information. I am going to ask
> on ppc64 mailing list on how to do it sanely, but I am afraid that
> they are going to say "all the information is available in the kernel
> data (lmb) structures, parse them - rather than exporting it
> to /proc/iomem". 
> 
> We may have to have arch-specific hooks to verify a memory region :(
> What do you think ?
> 
I think using IORESOURCE_MEM is better.
It is implemtend regardless of memory hotplug. I just reused it.
(I think x86's resouce struct is extened to 64bit for supporting memory info.)

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[Mark Gross]

On Tue, Oct 02, 2007 at 01:14:47AM +0900, KAMEZAWA Hiroyuki wrote:
> On Mon, 01 Oct 2007 08:37:43 -0700
> Badari Pulavarty <pbadari@gmail.com> wrote:
> > 1) Other than remove_memory(), I don't see any other arch-specific
> > code that needs to be provided. Even remove_memory() looks pretty
> > arch independent. Isn't it ?
> > 
> Yes, maybe arch independent. Current codes is based on assumption
> that some arch may needs some code before/after hotremove.
> If no arch needs, we can merge all. 
> 
> > 2) I copied remove_memory() from IA64 to PPC64. When I am testing
> > hotplug-remove (echo offline > state), I am not able to remove
> > any memory at all. I get different type of failures like ..
> > 
> > memory offlining 6e000 to 6f000 failed
> > 
> I'm not sure about this...does this memory is in ZONE_MOVABLE ?
> If not ZONE_MOVABLE, offlining can be fail because of not-removable
> kernel memory. 

How could I mark a nid's worth of memory as ZONE_MOVABLE?  I've been
reading through this code and it appears to somewhat arbitrarily choose
some portion of the memory to be ZONE_MOVABLE per pxm and some kernel
parameters.  But I'm having a hard time finding the proper place to set
up the nodes.

btw: I'm trying to finish up that power managed memory experiment where we
set up numa PXM's marking fbdims we want to fiddle with power state on.


--mgross

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[KAMEZAWA Hiroyuki]

On Fri, 5 Oct 2007 10:21:28 -0700
Mark Gross <mgross@linux.intel.com> wrote:

> > I'm not sure about this...does this memory is in ZONE_MOVABLE ?
> > If not ZONE_MOVABLE, offlining can be fail because of not-removable
> > kernel memory. 
> 
> How could I mark a nid's worth of memory as ZONE_MOVABLE?  I've been
> reading through this code and it appears to somewhat arbitrarily choose
> some portion of the memory to be ZONE_MOVABLE per pxm and some kernel
> parameters.  But I'm having a hard time finding the proper place to set
> up the nodes.
> 
It's not available now.

One idea is to ignore memory of some PXMs specified by kerenel boot param.
Later, you can hot-add specified PXM memory as MOVABLE.
Then, boot sequence will be
--
   bootstrap , ignore some memory here.
   init memory, driver, etc
   hot-add ignored memory
   online hot-added memory by user scripts. (from rc script ?)
--
For doing this, we need
 - a switch to hot-add memory as MOVABLE (will be easy patch)
 - a code for ignoring memory at boot but remember them for later hotadd
   (maybe needs arch specific codes)
 - a code for hot add memory before rc script (initcall is suitable ?) 

Needs some amount of arch-specific codes, but maybe simple.
Why I recommend above is it will be complex to avoid some PXM's memory
to be used as bootmem or for some other purpose(slab, hash, etc...).

If your firmware (efi?) doesn't show memory for hot removal at boot time,
this idea will be simpler..


Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[Mark Gross]

On Sat, Oct 06, 2007 at 09:41:15AM +0900, KAMEZAWA Hiroyuki wrote:
> On Fri, 5 Oct 2007 10:21:28 -0700
> Mark Gross <mgross@linux.intel.com> wrote:
> 
> > > I'm not sure about this...does this memory is in ZONE_MOVABLE ?
> > > If not ZONE_MOVABLE, offlining can be fail because of not-removable
> > > kernel memory. 
> > 
> > How could I mark a nid's worth of memory as ZONE_MOVABLE?  I've been
> > reading through this code and it appears to somewhat arbitrarily choose
> > some portion of the memory to be ZONE_MOVABLE per pxm and some kernel
> > parameters.  But I'm having a hard time finding the proper place to set
> > up the nodes.
> > 
> It's not available now.

Thats a usability challenge.  What use scenarios do you have for memory
unplug then?  I'd like to mimic your stuff if I can.

> 
> One idea is to ignore memory of some PXMs specified by kerenel boot param.
> Later, you can hot-add specified PXM memory as MOVABLE.
> Then, boot sequence will be
> --
>    bootstrap , ignore some memory here.
>    init memory, driver, etc
>    hot-add ignored memory
>    online hot-added memory by user scripts. (from rc script ?)
> --y
> For doing this, we need
>  - a switch to hot-add memory as MOVABLE (will be easy patch)
>  - a code for ignoring memory at boot but remember them for later hotadd
>    (maybe needs arch specific codes)
>  - a code for hot add memory before rc script (initcall is suitable ?) 
> 
> Needs some amount of arch-specific codes, but maybe simple.
> Why I recommend above is it will be complex to avoid some PXM's memory
> to be used as bootmem or for some other purpose(slab, hash, etc...).

I have the boot memory allocations or off-lineable-memory taken care of.
I can see how the above would work, but I worry that it feels a bit
hackish.   

BTW Is this how memory hot remove is expected to be used?

> 
> If your firmware (efi?) doesn't show memory for hot removal at boot time,
> this idea will be simpler..

how so?  

thanks,

--mgross

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[KAMEZAWA Hiroyuki]

On Mon, 8 Oct 2007 12:01:23 -0700
Mark Gross <mgross@linux.intel.com> wrote:
> > > How could I mark a nid's worth of memory as ZONE_MOVABLE?  I've been
> > > reading through this code and it appears to somewhat arbitrarily choose
> > > some portion of the memory to be ZONE_MOVABLE per pxm and some kernel
> > > parameters.  But I'm having a hard time finding the proper place to set
> > > up the nodes.
> > > 
> > It's not available now.
> 
> Thats a usability challenge.  What use scenarios do you have for memory
> unplug then?  I'd like to mimic your stuff if I can.
> 
For us (fujitsu), we'll have to implement node-unplug, but not yet.
If we can, we can use node-hotplug/node-unplug (of hotplugged node).
This can be used from our hardware console's GUI.
(For us, specifing hot-removable memory by node-id (at boot) is enough.)

I know there is requirements for removing memory which is available at boot
time. But I'm not sure that what information the firmware shows about
hot-removable memory now. 

Specifying MOVABLE zone in precie address range at boot time is ideal but
I'm afraid that it makes memory init code too complicated. Hmm...



> > 
> > One idea is to ignore memory of some PXMs specified by kerenel boot param.
> > Later, you can hot-add specified PXM memory as MOVABLE.
> > Then, boot sequence will be
> > --
> >    bootstrap , ignore some memory here.
> >    init memory, driver, etc
> >    hot-add ignored memory
> >    online hot-added memory by user scripts. (from rc script ?)
> > --y
> > For doing this, we need
> >  - a switch to hot-add memory as MOVABLE (will be easy patch)
> >  - a code for ignoring memory at boot but remember them for later hotadd
> >    (maybe needs arch specific codes)
> >  - a code for hot add memory before rc script (initcall is suitable ?) 
> > 
> > Needs some amount of arch-specific codes, but maybe simple.
> > Why I recommend above is it will be complex to avoid some PXM's memory
> > to be used as bootmem or for some other purpose(slab, hash, etc...).
> 
> I have the boot memory allocations or off-lineable-memory taken care of.
> I can see how the above would work, but I worry that it feels a bit
> hackish.   
> 
> BTW Is this how memory hot remove is expected to be used?
> 
I think above is a simple way to go. How about you ? > Goto-san.
I don't want to make memory-initialization too complicated.
 
> > If your firmware (efi?) doesn't show memory for hot removal at boot time,
> > this idea will be simpler..
> 
> how so?  
>
just hot-add hot-removable memory after boot by some scripts which affects 
the firmware.

I'm glad to hear question about making use of memory-hot-removal in these days :)

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hotplug memory remove

[Yasunori Goto]

> > > One idea is to ignore memory of some PXMs specified by kerenel boot param.
> > > Later, you can hot-add specified PXM memory as MOVABLE.
> > > Then, boot sequence will be
> > > --
> > >    bootstrap , ignore some memory here.
> > >    init memory, driver, etc
> > >    hot-add ignored memory
> > >    online hot-added memory by user scripts. (from rc script ?)
> > > --y
> > > For doing this, we need
> > >  - a switch to hot-add memory as MOVABLE (will be easy patch)
> > >  - a code for ignoring memory at boot but remember them for later hotadd
> > >    (maybe needs arch specific codes)
> > >  - a code for hot add memory before rc script (initcall is suitable ?) 
> > > 
> > > Needs some amount of arch-specific codes, but maybe simple.
> > > Why I recommend above is it will be complex to avoid some PXM's memory
> > > to be used as bootmem or for some other purpose(slab, hash, etc...).
> > 
> > I have the boot memory allocations or off-lineable-memory taken care of.
> > I can see how the above would work, but I worry that it feels a bit
> > hackish.   
> > 
> > BTW Is this how memory hot remove is expected to be used?
> > 
> I think above is a simple way to go. How about you ? > Goto-san.
> I don't want to make memory-initialization too complicated.

Hmm. At first, I agreed with you. But I'm a bit worry about it too now.

When movable-memory is specified by boot option, I guess something
wrong case will occur like followings.

  ex) Movable node is specified by boot-option as "movable_node=2" on 
      4 node box.

    1) boot up with 2 movable node and 2 non-movable node.
    2) remove those 2 nodes physically.
    3) system must be removed by some reasons.
    4) system can't boot up due to only 2 MOVABLE nodes!

It depends how "movable area" is specified. please take care.

Bye.

-- 
Yasunori Goto 


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>