[PATCH Bug fix] acpi, movablemem_map: node0 should always be unhotpluggable when using SRAT.

[PATCH Bug fix] acpi, movablemem_map: node0 should always be unhotpluggable when using SRAT.

[Tang Chen]

When using movablemem_map=acpi, always set node0 as unhotpluggable, otherwise
if all the memory is hotpluggable, the kernel will fail to boot.

When using movablemem_map=nn[KMG]@ss[KMG], we don't stop users specifying
node0 as hotpluggable, and ignore all the info in SRAT, so that this option
can be used as a workaround of firmware bugs.

Reported-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
---
 Documentation/kernel-parameters.txt |    6 ++++++
 arch/x86/mm/srat.c                  |   10 ++++++++--
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 7d1b6fc..81b6f15 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1645,6 +1645,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			in flags from SRAT from ACPI BIOS to determine which
 			memory devices could be hotplugged. The corresponding
 			memory ranges will be set as ZONE_MOVABLE.
+			NOTE: node0 should always be unhotpluggable, otherwise
+			      the kernel will fail to boot.
 
 	movablemem_map=nn[KMG]@ss[KMG]
 			[KNL,X86,IA-64,PPC] This parameter is similar to
@@ -1666,6 +1668,10 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			satisfied. So the administrator should be careful that
 			the amount of movablemem_map areas are not too large.
 			Otherwise kernel won't have enough memory to start.
+			NOTE: We don't stop users specifying node0 as
+			      hotpluggable, and ingore all the info in SRAT so
+			      that this option can be used as a workaround of
+			      firmware bugs.
 
 	MTD_Partition=	[MTD]
 			Format: <name>,<region-number>,<size>,<offset>
diff --git a/arch/x86/mm/srat.c b/arch/x86/mm/srat.c
index b20b5b7..a85d2b7 100644
--- a/arch/x86/mm/srat.c
+++ b/arch/x86/mm/srat.c
@@ -161,9 +161,13 @@ handle_movablemem(int node, u64 start, u64 end, u32 hotpluggable)
 	 *
 	 * Using movablemem_map, we can prevent memblock from allocating memory
 	 * on ZONE_MOVABLE at boot time.
+	 *
+	 * NOTE: node0 shoule always be unhotpluggable, otherwise, if all the
+	 *       memory is hotpluggable, there will be no memory kernel can use.
 	 */
 	if (hotpluggable && movablemem_map.acpi) {
-		insert_movablemem_map(start_pfn, end_pfn);
+		if (node != 0)
+			insert_movablemem_map(start_pfn, end_pfn);
 		goto out;
 	}
 
@@ -178,7 +182,9 @@ handle_movablemem(int node, u64 start, u64 end, u32 hotpluggable)
 	 * Using movablemem_map, we can prevent memblock from allocating memory
 	 * on ZONE_MOVABLE at boot time.
 	 *
-	 * NOTE: In this case, SRAT info will be ingored.
+	 * NOTE: We don't stop users specifying node0 as hotpluggable, and
+	 *       ignore all the info in SRAT, so that this option can be used
+	 *       as a workaround of firmware bugs.
 	 */
 	overlap = movablemem_map_overlap(start_pfn, end_pfn);
 	if (overlap >= 0) {
-- 
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH Bug fix] acpi, movablemem_map: node0 should always be unhotpluggable when using SRAT.

[David Rientjes]

On Wed, 30 Jan 2013, Tang Chen wrote:

> When using movablemem_map=acpi, always set node0 as unhotpluggable, otherwise
> if all the memory is hotpluggable, the kernel will fail to boot.
> 
> When using movablemem_map=nn[KMG]@ss[KMG], we don't stop users specifying
> node0 as hotpluggable, and ignore all the info in SRAT, so that this option
> can be used as a workaround of firmware bugs.
> 

Could you elaborate on the failure you're seeing?

I've booted the kernel many times without memory on a node 0.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH Bug fix] acpi, movablemem_map: node0 should always be unhotpluggable when using SRAT.

[Tang Chen]

On 01/30/2013 04:50 PM, David Rientjes wrote:
> On Wed, 30 Jan 2013, Tang Chen wrote:
>
>> When using movablemem_map=acpi, always set node0 as unhotpluggable, otherwise
>> if all the memory is hotpluggable, the kernel will fail to boot.
>>
>> When using movablemem_map=nn[KMG]@ss[KMG], we don't stop users specifying
>> node0 as hotpluggable, and ignore all the info in SRAT, so that this option
>> can be used as a workaround of firmware bugs.
>>
>
> Could you elaborate on the failure you're seeing?
>
> I've booted the kernel many times without memory on a node 0.
>

Hi David,

The failure I'm trying to fix is that if all the memory is hotpluggable, 
and user
specified movablemem_map, my code will set all the memory as 
ZONE_MOVABLE, and kernel
will fail to allocate any memory, and it will fail to boot.

But I'm sorry if I didn't answer your question. :)

Are you saying your memory is not on node0, and your physical address
0x0 is not on node0 ? And your /sys fs don't have a node0 interface, it is
node1 or something else ?

If so, I think I'd better find another way to fix this problem because 
node0 may not be
the first node on the system.

Thanks. :)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH Bug fix] acpi, movablemem_map: node0 should always be unhotpluggable when using SRAT.

[David Rientjes]

On Wed, 30 Jan 2013, Tang Chen wrote:

> The failure I'm trying to fix is that if all the memory is hotpluggable, and
> user
> specified movablemem_map, my code will set all the memory as ZONE_MOVABLE, and
> kernel
> will fail to allocate any memory, and it will fail to boot.
> 

I'm curious, do you have a dmesg of the failure?

Historically I've seen this panic as late as build_sched_domains() 
because of a bad mapping between pxms and apicids that assumes node 0 is 
online and results in node_distance() being inaccurate.  I'm not sure if 
you're even getting that far in boot?

> Are you saying your memory is not on node0, and your physical address
> 0x0 is not on node0 ? And your /sys fs don't have a node0 interface, it is
> node1 or something else ?
> 

Exactly, there is a node 0 but it includes no online memory (and that 
should be the case as if it was solely hotpluggable memory) at the time of 
boot.  The sysfs interfaces only get added if the memory is onlined later.

> If so, I think I'd better find another way to fix this problem because node0
> may not be
> the first node on the system.
> 

I haven't tried it over the past year or so, but this used to work in the 
past.  I think if we had some more information we'd be able to see if we 
really need to treat node 0 in a special way.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH Bug fix] acpi, movablemem_map: node0 should always be unhotpluggable when using SRAT.

[Tang Chen]

Hi David,

On 01/30/2013 05:45 PM, David Rientjes wrote:
> On Wed, 30 Jan 2013, Tang Chen wrote:
>
>> The failure I'm trying to fix is that if all the memory is hotpluggable, and
>> user
>> specified movablemem_map, my code will set all the memory as ZONE_MOVABLE, and
>> kernel
>> will fail to allocate any memory, and it will fail to boot.
>>
>
> I'm curious, do you have a dmesg of the failure?
>
> Historically I've seen this panic as late as build_sched_domains()
> because of a bad mapping between pxms and apicids that assumes node 0 is
> online and results in node_distance() being inaccurate.  I'm not sure if
> you're even getting that far in boot?

I'm sorry I cannot provide you any dmesg. I am using a remote machine 
and if
it failed to boot very early, it will redirect nothing to me.

So I think I didn't go that far.

>
>> Are you saying your memory is not on node0, and your physical address
>> 0x0 is not on node0 ? And your /sys fs don't have a node0 interface, it is
>> node1 or something else ?
>>
>
> Exactly, there is a node 0 but it includes no online memory (and that
> should be the case as if it was solely hotpluggable memory) at the time of
> boot.  The sysfs interfaces only get added if the memory is onlined later.

OK, you mean you have only node1 at first and no node0 interface, right?
If so, then this patch is wrong. :)

But you mean physical address 0x0 is on your node1, right? Otherwise, 
how could
the kernel be loaded ?

Could you provide the dmesg of your box like this:

[    0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x7fffffff]
[    0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x7ffffffff]
[    0.000000] SRAT: Node 1 PXM 2 [mem 0x1000000000-0x17ffffffff] Hot 
Pluggable
[    0.000000] SRAT: Node 2 PXM 3 [mem 0x1800000000-0x1fffffffff] Hot 
Pluggable
[    0.000000] SRAT: Node 3 PXM 4 [mem 0x2000000000-0x27ffffffff]
[    0.000000] SRAT: Node 4 PXM 5 [mem 0x2800000000-0x2fffffffff]
[    0.000000] SRAT: Node 5 PXM 6 [mem 0x3000000000-0x37ffffffff]
[    0.000000] SRAT: Node 6 PXM 7 [mem 0x3800000000-0x3fffffffff]
[    0.000000] SRAT: Node 7 PXM 1 [mem 0x800000000-0xfffffffff]

>
>> If so, I think I'd better find another way to fix this problem because node0
>> may not be
>> the first node on the system.
>>
>
> I haven't tried it over the past year or so, but this used to work in the
> past.  I think if we had some more information we'd be able to see if we
> really need to treat node 0 in a special way.
>
I'll try to do more investigation and find a better way to fix it. :)

Thanks. :)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH Bug fix] acpi, movablemem_map: node0 should always be unhotpluggable when using SRAT.

[H. Peter Anvin]

It is not node 0, it is whatever node the kernel resides in.

Tang Chen <tangchen@cn.fujitsu.com> wrote:

>When using movablemem_map=acpi, always set node0 as unhotpluggable,
>otherwise
>if all the memory is hotpluggable, the kernel will fail to boot.
>
>When using movablemem_map=nn[KMG]@ss[KMG], we don't stop users
>specifying
>node0 as hotpluggable, and ignore all the info in SRAT, so that this
>option
>can be used as a workaround of firmware bugs.
>
>Reported-by: H. Peter Anvin <hpa@zytor.com>
>Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
>---
> Documentation/kernel-parameters.txt |    6 ++++++
> arch/x86/mm/srat.c                  |   10 ++++++++--
> 2 files changed, 14 insertions(+), 2 deletions(-)
>
>diff --git a/Documentation/kernel-parameters.txt
>b/Documentation/kernel-parameters.txt
>index 7d1b6fc..81b6f15 100644
>--- a/Documentation/kernel-parameters.txt
>+++ b/Documentation/kernel-parameters.txt
>@@ -1645,6 +1645,8 @@ bytes respectively. Such letter suffixes can also
>be entirely omitted.
> 			in flags from SRAT from ACPI BIOS to determine which
> 			memory devices could be hotplugged. The corresponding
> 			memory ranges will be set as ZONE_MOVABLE.
>+			NOTE: node0 should always be unhotpluggable, otherwise
>+			      the kernel will fail to boot.
> 
> 	movablemem_map=nn[KMG]@ss[KMG]
> 			[KNL,X86,IA-64,PPC] This parameter is similar to
>@@ -1666,6 +1668,10 @@ bytes respectively. Such letter suffixes can
>also be entirely omitted.
> 			satisfied. So the administrator should be careful that
> 			the amount of movablemem_map areas are not too large.
> 			Otherwise kernel won't have enough memory to start.
>+			NOTE: We don't stop users specifying node0 as
>+			      hotpluggable, and ingore all the info in SRAT so
>+			      that this option can be used as a workaround of
>+			      firmware bugs.
> 
> 	MTD_Partition=	[MTD]
> 			Format: <name>,<region-number>,<size>,<offset>
>diff --git a/arch/x86/mm/srat.c b/arch/x86/mm/srat.c
>index b20b5b7..a85d2b7 100644
>--- a/arch/x86/mm/srat.c
>+++ b/arch/x86/mm/srat.c
>@@ -161,9 +161,13 @@ handle_movablemem(int node, u64 start, u64 end,
>u32 hotpluggable)
> 	 *
>	 * Using movablemem_map, we can prevent memblock from allocating
>memory
> 	 * on ZONE_MOVABLE at boot time.
>+	 *
>+	 * NOTE: node0 shoule always be unhotpluggable, otherwise, if all the
>+	 *       memory is hotpluggable, there will be no memory kernel can
>use.
> 	 */
> 	if (hotpluggable && movablemem_map.acpi) {
>-		insert_movablemem_map(start_pfn, end_pfn);
>+		if (node != 0)
>+			insert_movablemem_map(start_pfn, end_pfn);
> 		goto out;
> 	}
> 
>@@ -178,7 +182,9 @@ handle_movablemem(int node, u64 start, u64 end, u32
>hotpluggable)
>	 * Using movablemem_map, we can prevent memblock from allocating
>memory
> 	 * on ZONE_MOVABLE at boot time.
> 	 *
>-	 * NOTE: In this case, SRAT info will be ingored.
>+	 * NOTE: We don't stop users specifying node0 as hotpluggable, and
>+	 *       ignore all the info in SRAT, so that this option can be used
>+	 *       as a workaround of firmware bugs.
> 	 */
> 	overlap = movablemem_map_overlap(start_pfn, end_pfn);
> 	if (overlap >= 0) {

-- 
Sent from my mobile phone. Please excuse brevity and lack of formatting.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH Bug fix] acpi, movablemem_map: node0 should always be unhotpluggable when using SRAT.

[David Rientjes]

On Wed, 30 Jan 2013, Tang Chen wrote:

> > Exactly, there is a node 0 but it includes no online memory (and that
> > should be the case as if it was solely hotpluggable memory) at the time of
> > boot.  The sysfs interfaces only get added if the memory is onlined later.
> 
> OK, you mean you have only node1 at first and no node0 interface, right?
> If so, then this patch is wrong. :)
> 

Not usually unless I modify my SRAT or I start pulling DIMMs, but yes, 
I've booted the kernel many times in the past with no node 0 online.  As 
far as I know, there's no special casing that is needed for node 0 to 
assume it's online and I've tried to fix up places where that assumption 
has been made.  I'm sure that node_online_map must include at least one 
online node, obviously, but there should be no requirement that it be node 
0.

> But you mean physical address 0x0 is on your node1, right? Otherwise, how
> could
> the kernel be loaded ?
> 

Yes, the online pxms all point to nodes that do not have the node id of 0.

Is it possible to try earlyprintk and get a serial console connected or 
reproduce it locally to find out where the problem is?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>