From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id D2CAE8E0001 for ; Mon, 10 Dec 2018 07:37:43 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id t2so5198449edb.22 for ; Mon, 10 Dec 2018 04:37:43 -0800 (PST) Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id d13-v6si1503841ejt.105.2018.12.10.04.37.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Dec 2018 04:37:42 -0800 (PST) Date: Mon, 10 Dec 2018 13:37:38 +0100 From: Michal Hocko Subject: Re: [PATCH] mm/alloc: fallback to first node if the wanted node offline Message-ID: <20181210123738.GN1286@dhcp22.suse.cz> References: <20181206121152.GH1286@dhcp22.suse.cz> <20181207075322.GS1286@dhcp22.suse.cz> <20181207113044.GB1286@dhcp22.suse.cz> <20181207142240.GC1286@dhcp22.suse.cz> <20181207155627.GG1286@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20181207155627.GG1286@dhcp22.suse.cz> Sender: owner-linux-mm@kvack.org List-ID: To: Pingfan Liu Cc: Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Bjorn Helgaas , Jonathan Cameron On Fri 07-12-18 16:56:27, Michal Hocko wrote: > On Fri 07-12-18 22:27:13, Pingfan Liu wrote: > [...] > > diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c > > index 1308f54..4dc497d 100644 > > --- a/arch/x86/mm/numa.c > > +++ b/arch/x86/mm/numa.c > > @@ -754,18 +754,23 @@ void __init init_cpu_to_node(void) > > { > > int cpu; > > u16 *cpu_to_apicid = early_per_cpu_ptr(x86_cpu_to_apicid); > > + int node, nr; > > > > BUG_ON(cpu_to_apicid == NULL); > > + nr = cpumask_weight(cpu_possible_mask); > > + > > + /* bring up all possible node, since dev->numa_node */ > > + //should check acpi works for node possible, > > + for_each_node(node) > > + if (!node_online(node)) > > + init_memory_less_node(node); > > I suspect there is no change if you replace for_each_node by > for_each_node_mask(nid, node_possible_map) > > here. If that is the case then we are probably calling > free_area_init_node too early. I do not see it yet though. OK, so it is not about calling it late or soon. It is just that node_possible_map is a misnomer and it has a different semantic than I've expected. numa_nodemask_from_meminfo simply considers only nodes with some memory. So my patch didn't really make any difference and the node stayed uninialized. In other words. Does the following work? I am sorry to wildguess this way but I am not able to recreate your setups to play with this myself. diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c index 1308f5408bf7..d51643e10d00 100644 --- a/arch/x86/mm/numa.c +++ b/arch/x86/mm/numa.c @@ -216,8 +216,6 @@ static void __init alloc_node_data(int nid) node_data[nid] = nd; memset(NODE_DATA(nid), 0, sizeof(pg_data_t)); - - node_set_online(nid); } /** @@ -527,6 +525,19 @@ static void __init numa_clear_kernel_node_hotplug(void) } } +static void __init init_memory_less_node(int nid) +{ + unsigned long zones_size[MAX_NR_ZONES] = {0}; + unsigned long zholes_size[MAX_NR_ZONES] = {0}; + + free_area_init_node(nid, zones_size, 0, zholes_size); + + /* + * All zonelists will be built later in start_kernel() after per cpu + * areas are initialized. + */ +} + static int __init numa_register_memblks(struct numa_meminfo *mi) { unsigned long uninitialized_var(pfn_align); @@ -570,7 +581,7 @@ static int __init numa_register_memblks(struct numa_meminfo *mi) return -EINVAL; /* Finally register nodes. */ - for_each_node_mask(nid, node_possible_map) { + for_each_node(nid) { u64 start = PFN_PHYS(max_pfn); u64 end = 0; @@ -592,6 +603,10 @@ static int __init numa_register_memblks(struct numa_meminfo *mi) continue; alloc_node_data(nid); + if (!end) + init_memory_less_node(nid); + else + node_set_online(nid); } /* Dump memblock with node info and return. */ @@ -721,21 +736,6 @@ void __init x86_numa_init(void) numa_init(dummy_numa_init); } -static void __init init_memory_less_node(int nid) -{ - unsigned long zones_size[MAX_NR_ZONES] = {0}; - unsigned long zholes_size[MAX_NR_ZONES] = {0}; - - /* Allocate and initialize node data. Memory-less node is now online.*/ - alloc_node_data(nid); - free_area_init_node(nid, zones_size, 0, zholes_size); - - /* - * All zonelists will be built later in start_kernel() after per cpu - * areas are initialized. - */ -} - /* * Setup early cpu_to_node. * @@ -763,9 +763,6 @@ void __init init_cpu_to_node(void) if (node == NUMA_NO_NODE) continue; - if (!node_online(node)) - init_memory_less_node(node); - numa_set_node(cpu, node); } } -- Michal Hocko SUSE Labs