From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03029CDB482 for ; Thu, 19 Oct 2023 09:35:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74CCD8D0198; Thu, 19 Oct 2023 05:35:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FAFB8D0019; Thu, 19 Oct 2023 05:35:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C2B48D0198; Thu, 19 Oct 2023 05:35:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 47F438D0019 for ; Thu, 19 Oct 2023 05:35:35 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0A391C121C for ; Thu, 19 Oct 2023 09:35:35 +0000 (UTC) X-FDA: 81361703430.13.55D0842 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf29.hostedemail.com (Postfix) with ESMTP id 55F11120005 for ; Thu, 19 Oct 2023 09:35:32 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=axZ6C03s; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf29.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697708132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=igHNLXMaPt2DFCb90fBO7DqTwOi8PTYIGzfepUVRJyI=; b=qCjw5QcdzwSM8E2EMNXTLMzchyZJsgUetyzz486TH1iFSv4cEaACmv5+4wzmkv+jzgDccJ pX4GfR9DAQyqoFeCAHIXF6Br3E7WydJyXZFup764P4p12GrqQGSSVf+FjUEF0yk3qmAQui 0RQusWGizn6K7bUl7JJDepy/hfwtvPI= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=axZ6C03s; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf29.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697708132; a=rsa-sha256; cv=none; b=TfGJFNnAf50lrYkcCQukWdJtBWMWtnLT5J8HURdL9TvGyyH9WnjCN2Zfr0rLc39jDx+MR/ AXnwPsjsngpVnH0aDrgOHbaGpKDMKycmwsPVUDqAjPSH1IN+CZ7I6zgWpiZQjFkXDa4ooF 3F9vE8Bn2J7Q5vrB4eDsLo0kil8lPMc= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 561A861B51; Thu, 19 Oct 2023 09:35:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7A7ADC433C9; Thu, 19 Oct 2023 09:35:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1697708131; bh=uMWdtkngp6pIrB6Yzva+d5KcRLh0y1iTkblF4Qad//Y=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=axZ6C03sVf1qgnGcWKKR6zQ0cHUzzJMlVYCbue9/8bIEbrsU3h1WkH4OB94AjbMVz Yvt/vnxwj92dSRgJ0Ym1gP+wddHqFq00hgEuM69mXwC9z/Q2P8BhBvhlpb9CfehzQ1 /5PuygQFdmspxlt0+WY3NgwZM82M8PTfyLtA+lA9x9M7l11PBP73vS5PMP2RtpjpcL mMt+1iuRuuI6+1Cupg8d/4hfzRXfZ9IIBj2lAXhHzdtfFCb90mpzwUZuD3ls9GWTfo 4wxjFBGWXvOE9kvNWP3MxI8i5SGQFe6vxniw9pXgtvI8b6GBbvYNTVaAAQizH37hVa Zrho5Tp/AqBwA== Date: Thu, 19 Oct 2023 12:35:15 +0300 From: Mike Rapoport To: Ingo Molnar Cc: x86@kernel.org, Andrew Morton , Andy Lutomirski , Borislav Petkov , Dave Hansen , David Hildenbrand , "H. Peter Anvin" , Ingo Molnar , Michal Hocko , Peter Zijlstra , Qi Zheng , Thomas Gleixner , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v2] x86/mm: Drop 4MB restriction on minimal NUMA node memory size Message-ID: <20231019093515.GE2824@kernel.org> References: <20231017062215.171670-1-rppt@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 55F11120005 X-Stat-Signature: kw7jzinr6zijct7prt83cqm6djqbqcei X-Rspam-User: X-HE-Tag: 1697708132-836288 X-HE-Meta: U2FsdGVkX1/Roxc0hKbj347BgKsQ/GgWM4TPKMmU/4aHpil0+Srh/lgpxZgByuTa4olGmyLJS5XfDBmQGDLatCxbopQNxRWlQ8aD/ab566FF6dkHuk8ZXoCwMgXi7Y92ZGzus1Tl2Eg6XDKLbCe22bTfQvtbjSHgu1x30YsxTsixqWmlFnZwlKp25XK0CEUkr+uhTdzSUjjbx7GuuNpqt/VC4/AcxWe/AKl6GjnZpdI+aB/azef3pPRO/kvVgSzg5xqC8p+/7DRyHqii9y1VK9WD+fdWWc8stZRg2xwMtPoX10ictvFCVI08RXdI/PAex3JNt6VQtVN1b2VZAyk3eaUaSkEEVV4AW2069v9i1YQoMXPx5DSFVrqiXqBOO4VtWLDxHUy0/Np5aFrvh7X+QfgbfTH2bRFRq9Z9Bo1ScyfTy5ycANc5JSS4TxP7xeOLuSVdWtwAziRVJkK9FM6WsSFeJM2l2r6l9xJ5/QyTf4ReW5B0uUYXVyLclaYrqxZeYyCVnr91afXlXRrbq/PAogw5x109zzl+H9lcg+ZGD8BjBKD4kui/+3mDoHdh3tQV7YuD7KZKKVlPWyq7ZPL3aIwVIYU2qmDrI6oAcWJq8SPbHvtknuqHRG8AD3yFoxeLoFhwZfxIaP5GANpCPdkD52QKJOw/jXb71ixdEkK7rYlibZUXHVE8fIuVUq3DL0UXBSvMw/kJ9MXIfEP+5UVbnXyc/bRn98cXHPVIDvkJ8RlThVTSAg+BhcGD5LvMW+XCGgMSPUwpQUDwyJWW2X71s7/DDoywOp5IjjOqCzQf7fjC78+AavvanW8xERc6vJ54eIobIGQA6mwn8JmKLYziyK+AX/SvMvz7A7CQiItuUlKuKIPTa9wk1cZoALt3tD5qfy3nS+IniZrQL6nqIQu06PBeYpdupm5ARGaeJazuj+VidUQwZAoHhIvcUL1Lg+mXLWDLERBrMck1jUZQMYm MJ6d5y+R nJJAFlNdFTmy31PqnJgyXN54HkG738ggGkDmgMrIyofXqnqAqHA1weNfsZqNkrQnn1YgRVV9tD/9ZKQXi5bYZUJcclvCGJmAMLfASVG5AGZp/7ILJOxwM7NxAJGbBDV3m4Uwelu1FbTat1QSi7HgbzwXCGYVc5D/JMpUBUcV86/PMcohDpTl2ng1+awrkk08qOoyVNYU6woos8mkIy3fcThgr2BqR6iXvTQe5HwPQtqg3TBFaMnBdiEZAluArgVSNMhHw1UpuDEnXUmAyAohM+sZlI+g7scCoMMlbr/l8DK+VrJKk8w0dTkhDob7FN+mOCb2VYC5s8D1+Jf1g9sPcXj/wfsmFMXr43agXOdvwU24A4IkcDVfRHO8cwctcjq2TNiXQIkdER2R+egU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Oct 18, 2023 at 12:42:50PM +0200, Ingo Molnar wrote: > > * Mike Rapoport wrote: > > > From: "Mike Rapoport (IBM)" > > > > Qi Zheng reports crashes in a production environment and provides a > > simplified example as a reproducer: > > > > For example, if we use qemu to start a two NUMA node kernel, > > one of the nodes has 2M memory (less than NODE_MIN_SIZE), > > and the other node has 2G, then we will encounter the > > following panic: > > > > [ 0.149844] BUG: kernel NULL pointer dereference, address: 0000000000000000 > > [ 0.150783] #PF: supervisor write access in kernel mode > > [ 0.151488] #PF: error_code(0x0002) - not-present page > > <...> > > [ 0.156056] RIP: 0010:_raw_spin_lock_irqsave+0x22/0x40 > > <...> > > [ 0.169781] Call Trace: > > [ 0.170159] > > [ 0.170448] deactivate_slab+0x187/0x3c0 > > [ 0.171031] ? bootstrap+0x1b/0x10e > > [ 0.171559] ? preempt_count_sub+0x9/0xa0 > > [ 0.172145] ? kmem_cache_alloc+0x12c/0x440 > > [ 0.172735] ? bootstrap+0x1b/0x10e > > [ 0.173236] bootstrap+0x6b/0x10e > > [ 0.173720] kmem_cache_init+0x10a/0x188 > > [ 0.174240] start_kernel+0x415/0x6ac > > [ 0.174738] secondary_startup_64_no_verify+0xe0/0xeb > > [ 0.175417] > > [ 0.175713] Modules linked in: > > [ 0.176117] CR2: 0000000000000000 > > > > The crashes happen because of inconsistency between nodemask that has > > nodes with less than 4MB as memoryless and the actual memory fed into > > core mm. > > Presumably the core MM got fixed too to not just crash, but provide some > sort of warning? > > > The commit 9391a3f9c7f1 ("[PATCH] x86_64: Clear more state when ignoring > > empty node in SRAT parsing") that introduced minimal size of a NUMA node > > does not explain why a node size cannot be less than 4MB and what boot > > failures this restriction might fix. > > > > Since then a lot has changed and core mm won't confuse badly about small > > node sizes. > > Core MM won't get confused ... other than by the above weird Qemu topology, > to which it responds with a ... NULL pointer dereference? > > Seems quite close to the literal definition of 'get confused badly' to me, > and doesn't give me the warm fuzzy feeling that giving the core MM even > *more* weird topologies is super safe ... :-/ The confusion is not about topology and not because of the small size of a node. The confusion is because x86 fails to consistently report it's memory layout. At one point it says there is memoryless node and another point it says that that node actually has memory. And dropping the kludge that says "Don't confuse VM with a node that doesn't have the minimum amount of memory" fixes exactly that. > > Drop the limitation for the minimal node size. > > While I agree with dropping the limitation, and I agree that 9391a3f9c7f1 > should have provided more of a justification, I believe a core MM fix is in > order as well, for it to not crash. [ If it's fixed upstream already, > please reference the relevant commit ID. ] The core mm can be more strict about memory layouts it accepts indeed. I'll look into that. > Also, the changelog spelling & general presentation were quite low quality > - I've fixed it up a bit below, please carry this version going forward. > Please spell-check your patches before sending out Nth versions of it, > maybe maintainers are skipping them for a reason! > > Thanks, > > Ingo > > =================> > From: "Mike Rapoport (IBM)" > Date: Tue, 17 Oct 2023 09:22:15 +0300 > Subject: [PATCH] x86/mm: Drop 4MB restriction on minimal NUMA node memory size > > Qi Zheng reported crashes in a production environment and provided a > simplified example as a reproducer: > > | For example, if we use qemu to start a two NUMA node kernel, > | one of the nodes has 2M memory (less than NODE_MIN_SIZE), > | and the other node has 2G, then we will encounter the > | following panic: > | > | BUG: kernel NULL pointer dereference, address: 0000000000000000 > | <...> > | RIP: 0010:_raw_spin_lock_irqsave+0x22/0x40 > | <...> > | Call Trace: > | > | deactivate_slab() > | bootstrap() > | kmem_cache_init() > | start_kernel() > | secondary_startup_64_no_verify() > > The crashes happen because of inconsistency between the nodemask that > has nodes with less than 4MB as memoryless, and the actual memory fed > into the core mm. > > The commit: > > 9391a3f9c7f1 ("[PATCH] x86_64: Clear more state when ignoring empty node in SRAT parsing") > > ... that introduced minimal size of a NUMA node does not explain why > a node size cannot be less than 4MB and what boot failures this > restriction might fix. > > In the 17 years since then a lot has changed and core mm won't get > confused about small node sizes. > > Drop the limitation for the minimal node size. > > [ mingo: Improved changelog clarity. ] > > Reported-by: Qi Zheng > Signed-off-by: Mike Rapoport (IBM) > Not-Yet-Signed-off-by: Ingo Molnar > Acked-by: David Hildenbrand > Acked-by: Michal Hocko > Link: https://lore.kernel.org/all/20230212110305.93670-1-zhengqi.arch@bytedance.com/ > --- > arch/x86/include/asm/numa.h | 7 ------- > arch/x86/mm/numa.c | 7 ------- > 2 files changed, 14 deletions(-) > > diff --git a/arch/x86/include/asm/numa.h b/arch/x86/include/asm/numa.h > index e3bae2b60a0d..ef2844d69173 100644 > --- a/arch/x86/include/asm/numa.h > +++ b/arch/x86/include/asm/numa.h > @@ -12,13 +12,6 @@ > > #define NR_NODE_MEMBLKS (MAX_NUMNODES*2) > > -/* > - * Too small node sizes may confuse the VM badly. Usually they > - * result from BIOS bugs. So dont recognize nodes as standalone > - * NUMA entities that have less than this amount of RAM listed: > - */ > -#define NODE_MIN_SIZE (4*1024*1024) > - > extern int numa_off; > > /* > diff --git a/arch/x86/mm/numa.c b/arch/x86/mm/numa.c > index c01c5506fd4a..aa39d678fe81 100644 > --- a/arch/x86/mm/numa.c > +++ b/arch/x86/mm/numa.c > @@ -602,13 +602,6 @@ static int __init numa_register_memblks(struct numa_meminfo *mi) > if (start >= end) > continue; > > - /* > - * Don't confuse VM with a node that doesn't have the > - * minimum amount of memory: > - */ > - if (end && (end - start) < NODE_MIN_SIZE) > - continue; > - > alloc_node_data(nid); > } > -- Sincerely yours, Mike.