[RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[keith]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

  My numa hardware (IBM x445 summit based) supports hot-add memory. 
When this feature is enabled in the bios and booted as with CONFIG_NUMA
I get a memory range exposed by the SRAT/ACPI parsing.  This range
express the amount of memory the "could" be added to the system.  
  
  This chunk extends from the end of physical memory to the end of the
i386 address space.  If the following my physical memory is 0x2C0000. 

(From the boot messages)
Memory range 0x80000 to 0xC0000 (type 0x0) in proximity domain 0x01 enabled
Memory range 0x100000 to 0x2C0000 (type 0x0) in proximity domain 0x01 enabled
Memory range 0x2C0000 to 0x1000000 (type 0x0) in proximity domain 0x01 enabled and removable
  
  These memory ranges I believe to be valid according to what I know
about the SRAT and the ACPI 2.0c specs.  (I am not an ACPI expert please
correct me if I am wrong!)

  The numa KVA code used the node_start and node_end values (obtained
from the above memory ranges) to make it's lowmem reservations.  The
problem is that the lowmem area reserved is quite large.  It reserves
the entire a lmem_map large enough for 0x1000000 address space.  I don't
feel this is a great use of lowmem on my system :)

  Thankfully as we know the e820 shows what memory is really in the
system.  My simple fix it to find the max_pfn from the e820 earlier and
set the numa KVA areas accordingly. 
 
Don't trust the SRAT for this info only the e820.  

Thanks,
  Keith Mannthey 



[-- Attachment #2: fix2.patch --]
[-- Type: text/x-patch, Size: 1212 bytes --]

diff -urN linux-2.6.9/arch/i386/mm/discontig.c linux-2.6.9-fix2/arch/i386/mm/discontig.c
--- linux-2.6.9/arch/i386/mm/discontig.c	2004-11-16 17:41:18.207154544 -0800
+++ linux-2.6.9-fix2/arch/i386/mm/discontig.c	2004-11-16 17:37:05.811524512 -0800
@@ -199,6 +199,15 @@
 	unsigned long size, reserve_pages = 0;
 
 	for (nid = 1; nid < numnodes; nid++) {
+		/*
+		* The acpi/srat node info can show hot-add memroy zones
+		* where memory could be added but not currently present.
+		*/
+		if (node_start_pfn[nid] > max_pfn)
+			continue;
+		if (node_end_pfn[nid] > max_pfn)
+			node_end_pfn[nid] = max_pfn;
+
 		/* calculate the size of the mem_map needed in bytes */
 		size = (node_end_pfn[nid] - node_start_pfn[nid] + 1) 
 			* sizeof(struct page) + sizeof(pg_data_t);
@@ -261,12 +270,12 @@
 		printk("\n");
 	}
 
+	find_max_pfn();
 	reserve_pages = calculate_numa_remap_pages();
 
 	/* partially used pages are not usable - thus round upwards */
 	system_start_pfn = min_low_pfn = PFN_UP(init_pg_tables_end);
 
-	find_max_pfn();
 	system_max_low_pfn = max_low_pfn = find_max_low_pfn() - reserve_pages;
 	printk("reserve_pages = %ld find_max_low_pfn() ~ %ld\n",
 			reserve_pages, max_low_pfn + reserve_pages);

Re: [Lhms-devel] [RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[Dave Hansen]

On Tue, 2004-11-16 at 18:37, keith wrote:
>   The numa KVA code used the node_start and node_end values (obtained
> from the above memory ranges) to make it's lowmem reservations.  The
> problem is that the lowmem area reserved is quite large.  It reserves
> the entire a lmem_map large enough for 0x1000000 address space.  I don't
> feel this is a great use of lowmem on my system :)

It does seem silly to waste all of that lowmem for memory that *might*
be there, but what do you plan to do for contiguous address space (for
mem_map) once the memory addition occurs?  We've always talked about
having to preallocate mem_map space on 32-bit platforms and by your
patch it appears that this isn't what you want to do.  

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Lhms-devel] [RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[Yasunori Goto]

Hello, Keith-san.

>   This chunk extends from the end of physical memory to the end of the
> i386 address space.  If the following my physical memory is 0x2C0000. 
> 
> (From the boot messages)
> Memory range 0x80000 to 0xC0000 (type 0x0) in proximity domain 0x01 enabled
> Memory range 0x100000 to 0x2C0000 (type 0x0) in proximity domain 0x01 enabled
> Memory range 0x2C0000 to 0x1000000 (type 0x0) in proximity domain 0x01 enabled and removable
>   
>   These memory ranges I believe to be valid according to what I know
> about the SRAT and the ACPI 2.0c specs.  (I am not an ACPI expert please
> correct me if I am wrong!)

I also think this is valid. Probably the firmware of x445 thought, 
if enabled bit of SRAT is off, any other information of its area
will not be trusted.
So, there is no way to distinguish the machine can't attach more memory
from it can do it (just there is no memory AT BOOT TIME.)
So, third area in this boot message just indicates "possibility" of hotadd
memory.
But e820 probably indicates just memory areas which 
are already connected on the board, right?

(IIRC, there is no mention about if enable bit of SRAT is off
 in SRAT spec.)

BTW, I have a question.
  - Can x445 be attached memory without removing the node?
    In my concern machine, there is no physical space to
    hot add or exchange memory without physical removing
    the node. But, this SRAT table indicate that
    all of proximity is 0x01....
    Or is it just logical attachment?

Thanks.

-- 
Yasunori Goto <ygoto at us.fujitsu.com>


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Lhms-devel] [RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[Dave Hansen]

On Wed, 2004-11-17 at 14:33, Yasunori Goto wrote:
> But e820 probably indicates just memory areas which 
> are already connected on the board, right?

It's more than that.  It indicates which were connected the first time
that the machine was powered on.  If you suspend or hibernate the system
for some reason, it has to always present the e820 as it initially
appeared.  

> BTW, I have a question.
>   - Can x445 be attached memory without removing the node?
>     In my concern machine, there is no physical space to
>     hot add or exchange memory without physical removing
>     the node. But, this SRAT table indicate that
>     all of proximity is 0x01....
>     Or is it just logical attachment?

You can't remove nodes, just DIMMs.  The x440 hotplug is more like the
SMP case that I've always been concerned with.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Lhms-devel] [RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[Yasunori Goto]

> You can't remove nodes, just DIMMs.  The x440 hotplug is more like the
> SMP case that I've always been concerned with.

OK. I understood. 
Thanks.

-- 
Yasunori Goto <ygoto at us.fujitsu.com>


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Lhms-devel] [RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[keith]

On Wed, 2004-11-17 at 09:11, Dave Hansen wrote:
> On Tue, 2004-11-16 at 18:37, keith wrote:
> >   The numa KVA code used the node_start and node_end values (obtained
> > from the above memory ranges) to make it's lowmem reservations.  The
> > problem is that the lowmem area reserved is quite large.  It reserves
> > the entire a lmem_map large enough for 0x1000000 address space.  I don't
> > feel this is a great use of lowmem on my system :)
> 
> It does seem silly to waste all of that lowmem for memory that *might*
> be there, but what do you plan to do for contiguous address space (for
> mem_map) once the memory addition occurs?  We've always talked about
> having to preallocate mem_map space on 32-bit platforms and by your
> patch it appears that this isn't what you want to do.  
> 
> -- Dave

  I am not anticipating to support hot-add without config_nonlinear or
something similar which should provide more flexibility in allocation of
smaller section mem_maps.  This is only a issue when booted as a
discontig system.  We don't even consult the SRAT when we boot flat
(contiguous address space) so it is a non-issue.

  Wasting 500k of lowmem for memory that "might" be there is no good.  I
don't think having to preallocate the mem_map for a hot-add is really
that good.  What if the system never adds memory?  What if it only adds
8gig not 49g?  The system is crippled because it reserves the lmem_map
it "might" do a hot add with?  

  I forgot the mention that without this patch my system does not boot
with the hot-add support enabled in the bios.  

Thanks,
  Keith 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Lhms-devel] [RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[keith]

On Wed, 2004-11-17 at 14:33, Yasunori Goto wrote:
> Hello, Keith-san.
> 
> >   This chunk extends from the end of physical memory to the end of the
> > i386 address space.  If the following my physical memory is 0x2C0000. 
> > 
> > (From the boot messages)
> > Memory range 0x80000 to 0xC0000 (type 0x0) in proximity domain 0x01 enabled
> > Memory range 0x100000 to 0x2C0000 (type 0x0) in proximity domain 0x01 enabled
> > Memory range 0x2C0000 to 0x1000000 (type 0x0) in proximity domain 0x01 enabled and removable
> >   
> >   These memory ranges I believe to be valid according to what I know
> > about the SRAT and the ACPI 2.0c specs.  (I am not an ACPI expert please
> > correct me if I am wrong!)
> 
> I also think this is valid. Probably the firmware of x445 thought, 
> if enabled bit of SRAT is off, any other information of its area
> will not be trusted.
> So, there is no way to distinguish the machine can't attach more memory
> from it can do it (just there is no memory AT BOOT TIME.)
> So, third area in this boot message just indicates "possibility" of hotadd
> memory.
> But e820 probably indicates just memory areas which 
> are already connected on the board, right?

This is what I believe.  The e820 is the only true source of how much
memory is present in the system.  The SRAT just shows what the memory
layout is.  The SRAT is correct there is the possibility of memory being
in this zone.  

BTW: I can't toggle the SRAT enable bit in my bios.  I get it all the
time.    

> (IIRC, there is no mention about if enable bit of SRAT is off
>  in SRAT spec.)
> 
> BTW, I have a question.
>   - Can x445 be attached memory without removing the node?
>     In my concern machine, there is no physical space to
>     hot add or exchange memory without physical removing
>     the node. But, this SRAT table indicate that
>     all of proximity is 0x01....
>     Or is it just logical attachment?

My memory add works as follows.  I can physically add dimms in one of my
nodes.  I do not have to remove any node(cec) from my system.  Just open
up the case in insert dmmms into a empty memory bank.  

Thanks,
  Keith Mannthey   

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Lhms-devel] [RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[keith]

On Wed, 2004-11-17 at 14:42, Dave Hansen wrote:
> On Wed, 2004-11-17 at 14:33, Yasunori Goto wrote:
> > But e820 probably indicates just memory areas which 
> > are already connected on the board, right?
> 
> It's more than that.  It indicates which were connected the first time
> that the machine was powered on.  If you suspend or hibernate the system
> for some reason, it has to always present the e820 as it initially
> appeared.  

You use the acpi events to handle the addition of memory.  The e820 is
what you use to boot with. 

> > BTW, I have a question.
> >   - Can x445 be attached memory without removing the node?
> >     In my concern machine, there is no physical space to
> >     hot add or exchange memory without physical removing
> >     the node. But, this SRAT table indicate that
> >     all of proximity is 0x01....
> >     Or is it just logical attachment?
> 
> You can't remove nodes, just DIMMs.  The x440 hotplug is more like the
> SMP case that I've always been concerned with.
> -- Dave

My hardware only supports addition of memory not removal.  

Thanks,
  Keith Mannthey 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Lhms-devel] [RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[Dave Hansen]

On Wed, 2004-11-17 at 18:08, keith wrote:
>   I am not anticipating to support hot-add without config_nonlinear or
> something similar which should provide more flexibility in allocation of
> smaller section mem_maps.  This is only a issue when booted as a
> discontig system.  We don't even consult the SRAT when we boot flat
> (contiguous address space) so it is a non-issue.

Once a system has been running for any length of time, finding any
multi-order pages gets somewhat hard.  For a 16M section, you're still
talking about ~128k of mem_map, which is still an order 5 allocation. 
Nick's kswapd higher-order patches should help with this, though.

>   Wasting 500k of lowmem for memory that "might" be there is no good.  I
> don't think having to preallocate the mem_map for a hot-add is really
> that good.  What if the system never adds memory?  What if it only adds
> 8gig not 49g?  The system is crippled because it reserves the lmem_map
> it "might" do a hot add with?  

I have the feeling we'll eventually need a boot-time option for this
reservation.  Your patch, of course will work for now.  Do you want me
to pick it up in my tree?

>   I forgot the mention that without this patch my system does not boot
> with the hot-add support enabled in the bios.  

Why not?  I'm just curious what caused the actual failure.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [Lhms-devel] [RFC] fix for hot-add enabled SRAT/BIOS and numa KVA areas

[keith]

On Wed, 2004-11-17 at 18:24, Dave Hansen wrote:
> On Wed, 2004-11-17 at 18:08, keith wrote:
> >   I am not anticipating to support hot-add without config_nonlinear or
> > something similar which should provide more flexibility in allocation of
> > smaller section mem_maps.  This is only a issue when booted as a
> > discontig system.  We don't even consult the SRAT when we boot flat
> > (contiguous address space) so it is a non-issue.
> 
> Once a system has been running for any length of time, finding any
> multi-order pages gets somewhat hard.  For a 16M section, you're still
> talking about ~128k of mem_map, which is still an order 5 allocation. 
> Nick's kswapd higher-order patches should help with this, though.

I think it is safe to deal with this issue later.  I think i386 hot add
is going to be confined to a very small pool of boxes.  


> >   Wasting 500k of lowmem for memory that "might" be there is no good.  I
> > don't think having to preallocate the mem_map for a hot-add is really
> > that good.  What if the system never adds memory?  What if it only adds
> > 8gig not 49g?  The system is crippled because it reserves the lmem_map
> > it "might" do a hot add with?  
> 
> I have the feeling we'll eventually need a boot-time option for this
> reservation.  Your patch, of course will work for now.  Do you want me
> to pick it up in my tree?

I agree a boot time option to reserve a chunk of virtual address space
may be the way to go for i386.  I would hate to reserve too much space
that ends up just being wasted.  I think this can be save off for future
work.  I would like to get it into mm, I'll ping Andrew.    

> 
> >   I forgot the mention that without this patch my system does not boot
> > with the hot-add support enabled in the bios.  
> 
> Why not?  I'm just curious what caused the actual failure.
> 
> -- Dave

(snip)
Summit chipset: Starting Cyclone Counter.
Detected 1997.351 MHz processor.
Using cyclone for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Initializing HighMem for node 0
Initializing HighMem for node 1
Bad page state at free_hot_cold_page (in process 'swapper', page
d8e0d000)
flags:0xfffff7ff mapping:00000000 mapcount:0 count:266321921
Backtrace:
 [<c0140e09>] bad_page+0x81/0xb4
 [<c01415fd>] free_hot_cold_page+0x7e/0x100
 [<c0492849>] one_highpage_init+0xda/0x172
 [<c0493842>] set_highmem_pages_init+0xb0/0xf5
 [<c03691eb>] _etext+0x0/0x10d5
 [<c0492dcb>] mem_init+0x14f/0x2ba
 [<c0495c78>] alloc_large_system_hash+0x128/0x173
 [<c04827c7>] start_kernel+0x118/0x221
 [<c048231a>] unknown_bootoption+0x0/0x145
Trying to fix it up, but a reboot is needed
Bad page state at free_hot_cold_page (in process 'swapper', page
d8e0d020)
flags:0xfffff7ff mapping:00000000 mapcount:0 count:-523233279
Backtrace:
 [<c0140e09>] bad_page+0x81/0xb4
 [<c01415fd>] free_hot_cold_page+0x7e/0x100
 [<c0492849>] one_highpage_init+0xda/0x172
 [<c0493842>] set_highmem_pages_init+0xb0/0xf5
 [<c03691eb>] _etext+0x0/0x10d5
 [<c0492dcb>] mem_init+0x14f/0x2ba
 [<c0495c78>] alloc_large_system_hash+0x128/0x173
 [<c04827c7>] start_kernel+0x118/0x221
 [<c048231a>] unknown_bootoption+0x0/0x145
Trying to fix it up, but a reboot is needed
Bad page state at free_hot_cold_page (in process 'swapper', page
d8e0d040)
flags:0xfffff7ff mapping:00000000 mapcount:0 count:-523233279
Backtrace:
 [<c0140e09>] bad_page+0x81/0xb4
 [<c01415fd>] free_hot_cold_page+0x7e/0x100
 [<c0492849>] one_highpage_init+0xda/0x172

(and on and on and on)

Thanks,
  Keith Mannthey 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>