Re: [PATCH] mm/alloc: fallback to first node if the wanted node offline

[Michal Hocko <mhocko@kernel.org>]

On Thu 13-12-18 16:37:35, Pingfan Liu wrote:
[...]
> [    0.409667] NUMA: Node 1 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x7fffffff] -> [mem 0x00000000-0x7fffffff]
> [    0.419885] NUMA: Node 1 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x47fffffff] -> [mem 0x00000000-0x47fffffff]
> [    0.430386] NODE_DATA(0) allocated [mem 0x87efd4000-0x87effefff]
> [    0.436352]     NODE_DATA(0) on node 5
> [    0.440124] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000]
> [    0.447104] NODE_DATA(1) allocated [mem 0x47ffd5000-0x47fffffff]
> [    0.453110] NODE_DATA(2) allocated [mem 0x87efa9000-0x87efd3fff]
> [    0.459060]     NODE_DATA(2) on node 5
> [    0.462855] Initmem setup node 2 [mem 0x0000000000000000-0x0000000000000000]
> [    0.469809] NODE_DATA(3) allocated [mem 0x87ef7e000-0x87efa8fff]
> [    0.475788]     NODE_DATA(3) on node 5
> [    0.479554] Initmem setup node 3 [mem 0x0000000000000000-0x0000000000000000]
> [    0.486536] NODE_DATA(4) allocated [mem 0x87ef53000-0x87ef7dfff]
> [    0.492518]     NODE_DATA(4) on node 5
> [    0.496280] Initmem setup node 4 [mem 0x0000000000000000-0x0000000000000000]
> [    0.503266] NODE_DATA(5) allocated [mem 0x87ef28000-0x87ef52fff]
> [    0.509281] NODE_DATA(6) allocated [mem 0x87eefd000-0x87ef27fff]
> [    0.515224]     NODE_DATA(6) on node 5
> [    0.518987] Initmem setup node 6 [mem 0x0000000000000000-0x0000000000000000]
> [    0.525974] NODE_DATA(7) allocated [mem 0x87eed2000-0x87eefcfff]
> [    0.531953]     NODE_DATA(7) on node 5
> [    0.535716] Initmem setup node 7 [mem 0x0000000000000000-0x0000000000000000]

OK, so we have allocated node_data for all NUMA nodes. Good!

> [    0.542839] Reserving 500MB of memory at 384MB for crashkernel (System RAM: 32314MB)
> [    0.550465] Zone ranges:
> [    0.552927]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
> [    0.559081]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
> [    0.565235]   Normal   [mem 0x0000000100000000-0x000000087effffff]
> [    0.571388]   Device   empty
> [    0.574249] Movable zone start for each node
> [    0.578498] Early memory node ranges
> [    0.582049]   node   1: [mem 0x0000000000001000-0x000000000008efff]
> [    0.588291]   node   1: [mem 0x0000000000090000-0x000000000009ffff]
> [    0.594530]   node   1: [mem 0x0000000000100000-0x000000005c3d6fff]
> [    0.600772]   node   1: [mem 0x00000000643df000-0x0000000068ff7fff]
> [    0.607011]   node   1: [mem 0x000000006c528000-0x000000006fffffff]
> [    0.613251]   node   1: [mem 0x0000000100000000-0x000000047fffffff]
> [    0.619493]   node   5: [mem 0x0000000480000000-0x000000087effffff]
> [    0.626479] Zeroed struct page in unavailable ranges: 46490 pages
> [    0.626480] Initmem setup node 1 [mem 0x0000000000001000-0x000000047fffffff]
> [    0.655261] Initmem setup node 5 [mem 0x0000000480000000-0x000000087effffff]
[...]
> [    1.066324] Built 2 zonelists, mobility grouping off.  Total pages: 0

There are 2 zonelists built, but for some reason vm_total_pages is 0 and
that is clearly wrong.

Because the allocation failure (which later leads to NULL ptr) tells
there is quite a lot of memory.  One reason might be that the zonelist
for memory less nodes is initialized incorrectly. nr_free_zone_pages
relies on the local Node zonelist so if the code happened to run on a
cpu associated with Node2 then we could indeed got vm_total_pages=0.

> [    1.439440] Node 1 DMA: 2*4kB (U) 2*8kB (U) 2*16kB (U) 3*32kB (U) 2*64kB (U) 2*128kB (U) 2*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 15896kB
> [    1.453482] Node 1 DMA32: 2*4kB (M) 1*8kB (M) 1*16kB (M) 2*32kB (M) 3*64kB (M) 2*128kB (M) 3*256kB (M) 3*512kB (M) 2*1024kB (M) 3*2048kB (M) 255*4096kB (M) = 1055520kB
> [    1.468388] Node 1 Normal: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 1*1024kB (U) 1*2048kB (U) 31*4096kB (M) = 131068kB
> [    1.483211] Node 5 Normal: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 1*1024kB (U) 1*2048kB (U) 31*4096kB (M) = 131068kB

I am investigating what the hell is going on here. Maybe the former hack
to re-initialize memory-less nodes is working around some ordering
issues.
-- 
Michal Hocko
SUSE Labs