From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=3.0 tests=INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A054C433E0 for ; Wed, 1 Jul 2020 08:42:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D290920747 for ; Wed, 1 Jul 2020 08:42:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D290920747 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6A4718D0026; Wed, 1 Jul 2020 04:42:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F96B8D0024; Wed, 1 Jul 2020 04:42:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49AA98D0026; Wed, 1 Jul 2020 04:42:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0173.hostedemail.com [216.40.44.173]) by kanga.kvack.org (Postfix) with ESMTP id 2E5AD8D0024 for ; Wed, 1 Jul 2020 04:42:04 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D63102DFC for ; Wed, 1 Jul 2020 08:42:03 +0000 (UTC) X-FDA: 76988864526.17.crook48_57026a526e7f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id A9CEC180D0180 for ; Wed, 1 Jul 2020 08:42:03 +0000 (UTC) X-HE-Tag: crook48_57026a526e7f X-Filterd-Recvd-Size: 8370 Received: from mail-ej1-f67.google.com (mail-ej1-f67.google.com [209.85.218.67]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Wed, 1 Jul 2020 08:42:03 +0000 (UTC) Received: by mail-ej1-f67.google.com with SMTP id dr13so23670172ejc.3 for ; Wed, 01 Jul 2020 01:42:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=T9r9uXsGnf7xodRslPWJHaDGCw+J+NI5LCDeJSYGRKk=; b=iYRMcKU1TAn2g13uA/xEz/sYWec9pXL8vSWvzRBaJuMCcJKfVUTtRRCT5gxYhUQfm5 Qkeg2YFfYKSx/P1l+L19RDLFV6NOhWZVN4VYzOxPzHxPFITpkDBCnPloNDkH0Qy8eELI VToRCPgcdG2WVsRg0AyXCnYnHRIl3wJNk4oL1q1ZgvwlygJtbOmW5edFfj0Zsq5co/Yr 0I42DFwjyL+DnkNvMNIkkfG4gkJcMvDXN02rnbpAZFWfjQdG/NxhG02dWls3qRit9nga dBel38UgqbG9ENnggOE1iMIR//ga/9lVphXH8kbIJ0jkQY/zeZ9+CLyiuv7hRmfzPvA9 a4OA== X-Gm-Message-State: AOAM530CBJQCA1EogH9MUdepTm7R45LFer0jt2TmXdiKPtROANKS895p xI1KwMSeP7u8F19Zszet6xk= X-Google-Smtp-Source: ABdhPJyMVlYQgrE7cLMl4gjP0T0O//dKmjL8Y5w1eARBxVyjaPD/C6BUoqOwtCjSXGmrBV01flxkow== X-Received: by 2002:a17:906:abd6:: with SMTP id kq22mr22915729ejb.458.1593592922127; Wed, 01 Jul 2020 01:42:02 -0700 (PDT) Received: from localhost (ip-37-188-168-3.eurotel.cz. [37.188.168.3]) by smtp.gmail.com with ESMTPSA id b4sm5387790edx.96.2020.07.01.01.42.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2020 01:42:01 -0700 (PDT) Date: Wed, 1 Jul 2020 10:42:00 +0200 From: Michal Hocko To: Srikar Dronamraju Cc: Andrew Morton , linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Mel Gorman , Vlastimil Babka , "Kirill A. Shutemov" , Christopher Lameter , Michael Ellerman , Linus Torvalds , Gautham R Shenoy , Satheesh Rajendran , David Hildenbrand Subject: Re: [PATCH v5 3/3] mm/page_alloc: Keep memoryless cpuless node 0 offline Message-ID: <20200701084200.GN2369@dhcp22.suse.cz> References: <20200624092846.9194-1-srikar@linux.vnet.ibm.com> <20200624092846.9194-4-srikar@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200624092846.9194-4-srikar@linux.vnet.ibm.com> X-Rspamd-Queue-Id: A9CEC180D0180 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 24-06-20 14:58:46, Srikar Dronamraju wrote: > Currently Linux kernel with CONFIG_NUMA on a system with multiple > possible nodes, marks node 0 as online at boot. However in practice, > there are systems which have node 0 as memoryless and cpuless. > > This can cause numa_balancing to be enabled on systems with only one node > with memory and CPUs. The existence of this dummy node which is cpuless and > memoryless node can confuse users/scripts looking at output of lscpu / > numactl. > > By marking, N_ONLINE as NODE_MASK_NONE, lets stop assuming that Node 0 is > always online. > > v5.8-rc2 > available: 2 nodes (0,2) > node 0 cpus: > node 0 size: 0 MB > node 0 free: 0 MB > node 2 cpus: 0 1 2 3 4 5 6 7 > node 2 size: 32625 MB > node 2 free: 31490 MB > node distances: > node 0 2 > 0: 10 20 > 2: 20 10 > > proc and sys files > ------------------ > /sys/devices/system/node/online: 0,2 > /proc/sys/kernel/numa_balancing: 1 > /sys/devices/system/node/has_cpu: 2 > /sys/devices/system/node/has_memory: 2 > /sys/devices/system/node/has_normal_memory: 2 > /sys/devices/system/node/possible: 0-31 > > v5.8-rc2 + patch > ------------------ > available: 1 nodes (2) > node 2 cpus: 0 1 2 3 4 5 6 7 > node 2 size: 32625 MB > node 2 free: 31487 MB > node distances: > node 2 > 2: 10 > > proc and sys files > ------------------ > /sys/devices/system/node/online: 2 > /proc/sys/kernel/numa_balancing: 0 > /sys/devices/system/node/has_cpu: 2 > /sys/devices/system/node/has_memory: 2 > /sys/devices/system/node/has_normal_memory: 2 > /sys/devices/system/node/possible: 0-31 > > Note: On Powerpc, cpu_to_node of possible but not present cpus would > previously return 0. Hence this commit depends on commit ("powerpc/numa: Set > numa_node for all possible cpus") and commit ("powerpc/numa: Prefer node id > queried from vphn"). Without the 2 commits, Powerpc system might crash. > > 1. User space applications like Numactl, lscpu, that parse the sysfs tend to > believe there is an extra online node. This tends to confuse users and > applications. Other user space applications start believing that system was > not able to use all the resources (i.e missing resources) or the system was > not setup correctly. > > 2. Also existence of dummy node also leads to inconsistent information. The > number of online nodes is inconsistent with the information in the > device-tree and resource-dump > > 3. When the dummy node is present, single node non-Numa systems end up showing > up as NUMA systems and numa_balancing gets enabled. This will mean we take > the hit from the unnecessary numa hinting faults. I have to say that I dislike the node online/offline state and directly exporting that to the userspace. Users should only care whether the node has memory/cpus. Numa nodes can be online without any memory. Just offline all the present memory blocks but do not physically hot remove them and you are in the same situation. If users are confused by an output of tools like numactl -H then those could be updated and hide nodes without any memory&cpus. The autonuma problem sounds interesting but again this patch doesn't really solve the underlying problem because I strongly suspect that the problem is still there when a numa node gets all its memory offline as mentioned above. While I completely agree that making node 0 special is wrong, I have still hard time to review this very simply looking patch because all the numa initialization is so spread around that this might just blow up at unexpected places. IIRC we have discussed testing in the previous version and David has provided a way to emulate these configurations on x86. Did you manage to use those instruction for additional testing on other than ppc architectures? > Cc: linuxppc-dev@lists.ozlabs.org > Cc: linux-mm@kvack.org > Cc: linux-kernel@vger.kernel.org > Cc: Michal Hocko > Cc: Mel Gorman > Cc: Vlastimil Babka > Cc: "Kirill A. Shutemov" > Cc: Christopher Lameter > Cc: Michael Ellerman > Cc: Andrew Morton > Cc: Linus Torvalds > Cc: Gautham R Shenoy > Cc: Satheesh Rajendran > Cc: David Hildenbrand > Signed-off-by: Srikar Dronamraju > --- > Changelog v4:->v5: > - rebased to v5.8-rc2 > link v4: http://lore.kernel.org/lkml/20200512132937.19295-1-srikar@linux.vnet.ibm.com/t/#u > > Changelog v1:->v2: > - Rebased to v5.7-rc3 > Link v2: https://lore.kernel.org/linuxppc-dev/20200428093836.27190-1-srikar@linux.vnet.ibm.com/t/#u > > mm/page_alloc.c | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 48eb0f1410d4..5187664558e1 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -117,8 +117,10 @@ EXPORT_SYMBOL(latent_entropy); > */ > nodemask_t node_states[NR_NODE_STATES] __read_mostly = { > [N_POSSIBLE] = NODE_MASK_ALL, > +#ifdef CONFIG_NUMA > + [N_ONLINE] = NODE_MASK_NONE, > +#else > [N_ONLINE] = { { [0] = 1UL } }, > -#ifndef CONFIG_NUMA > [N_NORMAL_MEMORY] = { { [0] = 1UL } }, > #ifdef CONFIG_HIGHMEM > [N_HIGH_MEMORY] = { { [0] = 1UL } }, > -- > 2.18.1 > -- Michal Hocko SUSE Labs