32/64-bit NUMA consolidation behavior regresion

32/64-bit NUMA consolidation behavior regresion

[Dave Hansen]

Hi Tejun,

I was bisecting a boot problem on a 32-bit NUMA kernel and it bisected
down to commit 8db78cc4.  It turns out that, with this patch,
pcpu_need_numa() changed its return value on my system from 1 to 0.
What that basically meant was that we stopped using the remapped lowmem
areas for percpu data.

My system is just qemu booted with:

-smp 8 -m 8192 -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7

Watch the "PERCPU:" line early in boot, and you can see the "Embedded"
come and go with or without your patch:

[    0.000000] PERCPU: Embedded 11 pages/cpu @f3000000 s30592 r0 d14464
vs
[    0.000000] PERCPU: 11 4K pages/cpu @f83fe000 s30592 r0 d14464

I believe this has to do with the hunks in your patch that do:

-#ifdef CONFIG_X86_64
        init_cpu_to_node();
-#endif
...
-#ifdef CONFIG_X86_32
-DEFINE_EARLY_PER_CPU(int, x86_cpu_to_node_map, 0);
-#else
 DEFINE_EARLY_PER_CPU(int, x86_cpu_to_node_map, NUMA_NO_NODE);
-#endif
 EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_node_map);

I don't have a fix handy because I'm working on the original problem,
but I just happened to run across this during a bisect.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 32/64-bit NUMA consolidation behavior regresion

[Tejun Heo]

Hello, Dave.

On Wed, Nov 28, 2012 at 04:03:58PM -0800, Dave Hansen wrote:
> My system is just qemu booted with:
> 
> -smp 8 -m 8192 -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7
> 
> Watch the "PERCPU:" line early in boot, and you can see the "Embedded"
> come and go with or without your patch:
> 
> [    0.000000] PERCPU: Embedded 11 pages/cpu @f3000000 s30592 r0 d14464
> vs
> [    0.000000] PERCPU: 11 4K pages/cpu @f83fe000 s30592 r0 d14464
...
> I don't have a fix handy because I'm working on the original problem,
> but I just happened to run across this during a bisect.

Just tested 3.7-rc7 w/ qemu and it works as expected here.

Can you please boot with the following debug patch and report the boot
message before and after?

Thanks.

diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c
index 5cdff03..1133dc8 100644
--- a/arch/x86/kernel/setup_percpu.c
+++ b/arch/x86/kernel/setup_percpu.c
@@ -71,9 +71,13 @@ static bool __init pcpu_need_numa(void)
 	for_each_possible_cpu(cpu) {
 		int node = early_cpu_to_node(cpu);
 
+		printk("XXX pcpu_need_numa: cpu%d@%d online=%d ND=%p\n",
+		       cpu, node, node_online(node), NODE_DATA(node));
 		if (node_online(node) && NODE_DATA(node) &&
-		    last && last != NODE_DATA(node))
+		    last && last != NODE_DATA(node)) {
+			printk("XXX need numa\n");
 			return true;
+		}
 
 		last = NODE_DATA(node);
 	}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: 32/64-bit NUMA consolidation behavior regresion

[Dave Hansen]

On 11/30/2012 12:42 PM, Tejun Heo wrote:
> On Wed, Nov 28, 2012 at 04:03:58PM -0800, Dave Hansen wrote:
>> My system is just qemu booted with:
>>
>> -smp 8 -m 8192 -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7
>>
>> Watch the "PERCPU:" line early in boot, and you can see the "Embedded"
>> come and go with or without your patch:
>>
>> [    0.000000] PERCPU: Embedded 11 pages/cpu @f3000000 s30592 r0 d14464
>> vs
>> [    0.000000] PERCPU: 11 4K pages/cpu @f83fe000 s30592 r0 d14464
> ...
>> I don't have a fix handy because I'm working on the original problem,
>> but I just happened to run across this during a bisect.
> 
> Just tested 3.7-rc7 w/ qemu and it works as expected here.
> 
> Can you please boot with the following debug patch and report the boot
> message before and after?

Hi Tejun,

I just tested with 3.7-rc7 and I'm seeing the expected behavior now.
Looks like it got fixed along the way somewhere.  I was bisecting way
back in the 2.6.3x's.  Sorry of the noise.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>