From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18D30C47DDC for ; Tue, 23 Jan 2024 07:34:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCD076B0087; Tue, 23 Jan 2024 02:34:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D7E976B0089; Tue, 23 Jan 2024 02:34:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD06F6B008A; Tue, 23 Jan 2024 02:34:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A90B96B0087 for ; Tue, 23 Jan 2024 02:34:48 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4654BC046F for ; Tue, 23 Jan 2024 07:34:48 +0000 (UTC) X-FDA: 81709763856.08.5AD61DA Received: from mail.loongson.cn (mail.loongson.cn [114.242.206.163]) by imf10.hostedemail.com (Postfix) with ESMTP id 69A74C0026 for ; Tue, 23 Jan 2024 07:34:44 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of huangpei@loongson.cn designates 114.242.206.163 as permitted sender) smtp.mailfrom=huangpei@loongson.cn ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705995286; a=rsa-sha256; cv=none; b=I4hBY0kwc2xdo4xPwOVPJzqcJkrspb/khSEGQdRjmy3NOlnBuQaCZH2XEDwqgx1EdKd+4t PcyM/8Kd51jG0F7F7xfSoMmoXZGs19IBaEQ4fkvXXDSoxH+0ncU2gvheXagdwbmaVngqzk 11kwYHFlSWFniZaRXjgxYqaWMDPuDY8= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of huangpei@loongson.cn designates 114.242.206.163 as permitted sender) smtp.mailfrom=huangpei@loongson.cn ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705995286; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AtCzlQv5a1pDgRDF5wkeFc/NVOJOPrC9kdbXvaJBbPo=; b=1+yYIL/Pcfw7o0SCnNf6d8kMVGwbuXCz1OA5W9UYC2rl7C9Yd8OnWO1SE6Fp+VtWU6mvNI Y37XLLP1EcIV9dWJOhqhBio/fxLy2tJ7RRdZjgQ9OtejaS3poRT9olH/HQ9Ign42Z8MPxS mlHB39UJuK8LbkE7WEqRbRd9uIWAih8= Received: from loongson.cn (unknown [111.9.175.10]) by gateway (Coremail) with SMTP id _____8BxefBDjKhlfHMBAA--.7451S3; Thu, 18 Jan 2024 10:26:11 +0800 (CST) Received: from Board-3A3000 (unknown [111.9.175.10]) by localhost.localdomain (Coremail) with SMTP id AQAAf8DxzM5AjKhlUnwHAA--.1588S2; Thu, 18 Jan 2024 10:26:10 +0800 (CST) Date: Thu, 18 Jan 2024 10:26:08 +0800 From: Huang Pei To: Mike Rapoport Cc: Yajun Deng , Jiaxun Yang , linux-mm@kvack.org, Bibo Mao , linux-mips@vger.kernel.org, Paul Burton , Li Xuefeng , Yang Tiezhu , Gao Juxin , Huacai Chen , Thomas Bogendoerfer , linux-kernel@vger.kernel.org Subject: Re: memblock_reserve for unadded region (was: [PATCH] MIPS: loongson64: fix boot failure) Message-ID: <20240118022608.m4dzunfiogkdoazi@Board-3A3000> References: <731134fd-4b3d-418c-84ee-80646bffcc01@flygoat.com> <20240116122304.qwzy7san2vgspt2x@Board-3A3000> <3fc2f75e-d163-1ad1-009a-0e4538011885@linux.dev> <20240117030113.gs2fjs6vydthsc6l@Board-3A3000> <234ddaeb-5988-858d-6ba1-0fef90fb7a87@linux.dev> <20240117035910.453ofvotbvkzje6k@Board-3A3000> <20240117074546.gm7lejjt7whjtwgr@Board-3A3000> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-CM-TRANSID:AQAAf8DxzM5AjKhlUnwHAA--.1588S2 X-CM-SenderInfo: xkxd0whshlqz5rrqw2lrqou0/ X-Coremail-Antispam: 1Uk129KBj93XoWfJr18XFyDur1kGrWUAF1rXwc_yoWkGF48pr 1fJF17CrW8Jr18Jr4Utr15Jr1jyw1YyF1UXr1DJr18Xr1qyr13Xr1UJr18WFyDJrW8Jr12 qr1UJr12vr1UAwcCm3ZEXasCq-sJn29KB7ZKAUJUUUU5529EdanIXcx71UUUUU7KY7ZEXa sCq-sGcSsGvfJ3Ic02F40EFcxC0VAKzVAqx4xG6I80ebIjqfuFe4nvWSU5nxnvy29KBjDU 0xBIdaVrnRJUUUyEb4IE77IF4wAFF20E14v26r1j6r4UM7CY07I20VC2zVCF04k26cxKx2 IYs7xG6rWj6s0DM7CIcVAFz4kK6r1Y6r17M28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48v e4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_JFI_Gr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI 0_Gr0_Cr1l84ACjcxK6I8E87Iv67AKxVWxJVW8Jr1l84ACjcxK6I8E87Iv6xkF7I0E14v2 6r4UJVWxJr1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqjxCEc2xF0cIa020Ex4CE44I27w Aqx4xG64xvF2IEw4CE5I8CrVC2j2WlYx0E2Ix0cI8IcVAFwI0_Jrv_JF1lYx0Ex4A2jsIE 14v26r1j6r4UMcvjeVCFs4IE7xkEbVWUJVW8JwACjcxG0xvEwIxGrwCF04k20xvY0x0EwI xGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480 Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7 IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Jr0_Gr1lIxAIcVCF04k2 6cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxV AFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU8zwZ7UUUUU== X-Gw-Check: 325535f2152225ab X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 69A74C0026 X-Stat-Signature: bf8kjisymqbxaqkwj48gjtk789dz7ogi X-Rspam-User: X-HE-Tag: 1705995284-61169 X-HE-Meta: U2FsdGVkX19wkkRq/wpcDsrIs6cSFbEaXJMXbrdz4UTIVCQbJsttPc2KIW5WGWqx490QpEIMOuvjg1mMSuLIrREFghnH7Nx2k7tUFn45lXZ0E0sWTuk/jcAgjpwrbyYn6R/ARtb9oUI+wJoNq2B97xhq90Ng4E501hhcOcb1dKfZRfzcY5WjT05ZkfzQDMJ6joNeJO1jKm3iDi2Zv2igxcmDcWSbJk6kAqyxUSEnpndUugxyRa6asQ2J+Fr8LrTlo0hlilShHDY28ZAKc2r8dhZGYdzHTadNiUyQMpRvwiVbdToikwGbkBO2So64vNY2r2VyM7Ai+6PCEyTlD6KIwCGVLidnvk3/kvBkTRRfhpAl2HQ6eoQX1Mnemi07pZE76vGyVCApe6L6EoXtU3I+tdShF1dR1ZNNWu1dXefJgPpxoYGoiAQ/TEoFB1Y1eHB4EpAegaOUOeRY3BH52e4XVxBZ3gxIlGbalzgVq2Kz5ADgbW01DxnuHiH1+RRPHx4+WBoVdIpIXBxiitWD4Nq3eAB4kWoFi290MowcIAhQlXDYFk1+or2HXb1i7qWTsQjiEutNaIXWJGuF2uzRnwjZbvZNi0n87oycHjpJjxcIqexoZ+0nk4XQTT2BE62T4aKzG8bGJbHC58gJUvv+w77RhY06kmru3KgWfDZPUohu25pq26YO4n1+GLYuTULQunVhBOWRis388/r2l0Cpt1E9PW9lY/iSWFyEH0YjoEn2m+Ww7dKoVVHjjVCfGMxp+PWJeKO+7NlHTChAybbfUFXthhKot8dFWuYO4HQVRCuOpSEJaoEevvZWsz9K8KNJlbQqxg/bj0iT7cdcoTQy4pTaQarVKAmUgTdtxOdWI+BGssa3nxPVCprb8mJJBOpKu2jm+MyVbpsHdhb383OqFcg1mKrZW2IoGgQ77in4FM6IhodeCISgBWMtXrZvAXuqA8cVT8M1p17bAeS6mNGNvEg Q79q+2vb dak5v7xXR4xevD84AS3yEjfd3+G7syaurBoHzN0nLPpji48JAZInIZZLym83rSpZX7w3XOaOHrdyfGtw53PLX7jmCNQm5BXZG6x1u2aLbmcK2pmN+m4pZR1wLhiNNFjC57qvoBzLSdKr+ld6m4z2zHablwByR0Ern3AmlVFTqvWLqrIB13K9EwZzDqg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 17, 2024 at 01:08:48PM +0200, Mike Rapoport wrote: > On Wed, Jan 17, 2024 at 03:45:46PM +0800, Huang Pei wrote: > > On Wed, Jan 17, 2024 at 08:46:34AM +0200, Mike Rapoport wrote: > > > On Wed, Jan 17, 2024 at 11:59:10AM +0800, Huang Pei wrote: > > > > On Wed, Jan 17, 2024 at 11:17:18AM +0800, Yajun Deng wrote: > > > > > > > > > > On 2024/1/17 11:01, Huang Pei wrote: > > > > > > On Wed, Jan 17, 2024 at 10:20:00AM +0800, Yajun Deng wrote: > > > > > > > On 2024/1/16 20:23, Huang Pei wrote: > > > > > > > > On Tue, Jan 16, 2024 at 10:39:04AM +0200, Mike Rapoport wrote: > > > > > > > > > On Mon, Jan 15, 2024 at 02:08:21PM +0000, Jiaxun Yang wrote: > > > > > > > > > > Hi mm folks, > > > > > > > > > > > > > > > > > > > > Just a quick question, what is the expected behavior of memblock_reserve > > > > > > > > > > a region that is not added to memblock with memblock_add? > > > > > > > > > > > > > > > > > > > > I'm unable to find any documentation about memblock_reserve in comments and > > > > > > > > > > boot-time-mm, but as per my understanding to the code, this should be a > > > > > > > > > > legit usage? > > > > > > > > > Yes, memblock allows reserving memory that was not added to memblock with > > > > > > > > > memblock_add(). > > > > > > > > I think arch/platform specific code should fix this bug, like, > > > > > > > > -------------------------------------------------------------------------- > > > > > > > > //for loongson64 > > > > > > > > memblock_set_node(0, 1ULL << 44, &memblock.reserved, 0); > > > > > > > > > > > > > > > > -------------------------------------------------------------------------- > > > > > > > > > > > > > > > > or maybe memblock provide something like memblock_reserve_node > > > > > > > Hi pei, > > > > > > > > > > > > > > Can you test the following patch to see if it fixes this bug? > > > > > > > > > > > > > > diff --git a/mm/mm_init.c b/mm/mm_init.c > > > > > > > index 2c19f5515e36..97721d99fdce 100644 > > > > > > > --- a/mm/mm_init.c > > > > > > > +++ b/mm/mm_init.c > > > > > > > @@ -708,6 +708,9 @@ static void __meminit init_reserved_page(unsigned long > > > > > > > pfn, int nid) > > > > > > >         pg_data_t *pgdat; > > > > > > >         int zid; > > > > > > > > > > > > > > +       if (unlikely(nid == NUMA_NO_NODE || nid >= MAX_NUMNODES)) > > > > > > > +               nid = early_pfn_to_nid(pfn); > > > > > > > + > > > > > > >         if (early_page_initialised(pfn, nid)) > > > > > > >                 return; > > > > > > IMO this will fix the bug. Before 61167ad5fecd we had nid = first_online_node > > > for reserved pages that didn't have nid set in memblock.reserved. After > > > 61167ad5fecd we try to initialize these pages with MAX_NUMNODES and > > > obviously crash when accessing node structure. > > > > > > I think that the check for a valid nid should be moved to > > > memmap_init_reserved_pages() though. An entire reserved region will have > > > nid set to MAX_NUMNODES, so there is no point to check every page in it. > > > > > > > > > I do not think this fix set the right nid, ONLY arch/platform know that > > > > > > nid > > > > > > Why does it matter to have the right nid in a reserved page that is not > > > part of usable memory? > > > > > IMO, if a reserved page DO have a valid nid, and archs knows that, archs > > should set it right, and this is just the case of loongson64(and > > loongarch). I will set the nid for reserved page on loongson64, just like what loongarch did. > An arch may choose to set nids for reserved regions that are never added to > memblock.memory, but mm shouldn't crash if it didn't. > I agree. > > > That's true that only arch knows on which node those reserved pages are, > > > but core mm does not care about reserved pages that are not in memory. > > > > > > > > > int __meminit early_pfn_to_nid(unsigned long pfn) > > > > > > { > > > > > > static DEFINE_SPINLOCK(early_pfn_lock); > > > > > > int nid; > > > > > > > > > > > > spin_lock(&early_pfn_lock); > > > > > > nid = __early_pfn_to_nid(pfn, > > > > > > &early_pfnnid_cache); > > > > > > if (nid < 0) > > > > > > //!!!first_online_node MAY NOT be the node the pfn belong to!!! > > > > > > nid = first_online_node; > > > > > > > > > > > > spin_unlock(&early_pfn_lock); > > > > > > > > > > > > return > > > > > > nid; > > > > > > } > > > > > > > > > > > > > > > Okay, I don't think this bug is caused by commit 61167ad5fecd ("mm: pass nid > > > > > to reserve_bootmem_region()"), > > > > > > > > > > because even if you revert this commit, it will still get nid by > > > > > early_pfn_to_nid(). Did I get that right? > > > > > > > > Yes, more accurately, this bug is exposed by commit 61167ad5fecd. My > > > > previous fix is based on presumptions that memory_reserve should reserve memory > > > > added by memblock_add{,_node}, if going across this limitation, there need > > > > to set the valid nid for reserved memory region. > > > > > > > > > > > > > > > > > > > > > > In practical we run into uninitialized nid of reserved block problem, should > > > > > > > > > > we fix it > > > > > > > > > > in our usage, or on memblock side? > > > > > > > > > Apparently it's a bug in memblock :( > > > > > > > > > > > > > > > > > > If you revert 61167ad5fecd ("mm: pass nid to reserve_bootmem_region()") > > > > > > > > > does the issue disappear? > > > > > > > > Yes, I git bisect this commit. > > > > > > > > > > > > > > > > But I don't think it is a bug in memblock. IMO, memblock_reserve under > > > > > > > > NUMA set nid of reserved region to MAX_NUMNODES, which is the point > > > > > > > > that cause the "memblock_get_region_node from memmap_init_reserved_pages " > > > > > > > > passing a invalid node id(aka MAX_NUMNODES) to "reserver_bootmem_region > > > > > > > > -> init_reserved_page -> early_pfn_to_nid". If arch-specific code DOES NOT > > > > > > > > initialize the nid of reserved region(only it know that), or the reserved > > > > > > > > region NOT within a memblock added by memblock_add, memblock can not > > > > > > > > give a valid node id to the reserved region. Commit 61167ad5fecd ("mm: pass nid to > > > > > > > > reserve_bootmem_region()") just reveals the embarrassment case by an > > > > > > > > out of bound memory access. > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > 在 2023/12/25 09:30, Huang Pei 写道: > > > > > > > > > > > Since commit 61167ad5fecd("mm: pass nid to reserve_bootmem_region()), > > > > > > > > > > > loongson64 booting failed with CONFIG_DEFERRED_STRUCT_PAGE_INIT like > > > > > > > > > > > this: > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > > > > > Call Trace: > > > > > > > > > > > [] reserve_bootmem_region+0xa8/0x184 > > > > > > > > > > > [] memblock_free_all+0x104/0x2a8 > > > > > > > > > > > [] mem_init+0x84/0x94 > > > > > > > > > > > [] mm_core_init+0xf8/0x308 > > > > > > > > > > > [] start_kernel+0x43c/0x86c > > > > > > > > > > > > > > > > > > > > > > Code: 10400028 2402fff0 de420000 0203182b 14600022 > > > > > > > > > > > 64420070 00003025 24040003 > > > > > > > > > > > > > > > > > > > > > > ---[ end trace 0000000000000000 ]--- > > > > > > > > > > > Kernel panic - not syncing: Attempted to kill the idle task! > > > > > > > > > > > ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]--- > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > > > > > > > > > > > > > > > > The root cause is no memory region "0x0-0x1fffff" paired with > > > > > > > > > > > memory-reserved region "0x0-0x1fffff" and "0x0-0xfff", with "memblock > > > > > > > > > > > =debug": > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > > > > > memory[0x0] [0x0000000000200000-0x000000000effffff], > > > > > > > > > > > 0x000000000ee00000 bytes on node 0 flags: 0x0 !!!!here > > > > > > > > > > > memory[0x1] [0x0000000090000000-0x00000000fdffffff], > > > > > > > > > > > 0x000000006e000000 bytes on node 0 flags: 0x0 > > > > > > > > > > > memory[0x2] [0x0000000100000000-0x000000027fffffff], > > > > > > > > > > > 0x0000000180000000 bytes on node 0 flags: 0x0 > > > > > > > > > > > memory[0x3] [0x0000100000000000-0x000010000fffffff], > > > > > > > > > > > 0x0000000010000000 bytes on node 1 flags: 0x0 > > > > > > > > > > > memory[0x4] [0x0000100090000000-0x000010027fffffff], > > > > > > > > > > > 0x00000001f0000000 bytes on node 1 flags: 0x0 > > > > > > > > > > > reserved.cnt = 0x1f > > > > > > > > > > > reserved[0x0] [0x0000000000000000-0x000000000190c80a], > > > > > > > > > > > 0x000000000190c80b bytes flags: 0x0 !!!!oops 0x0-0x1fffff not in memory[0] > > > > > > > > > > > reserved[0x1] [0x000000000190c810-0x000000000190eea3], > > > > > > > > > > > 0x0000000000002694 bytes flags: 0x0 > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > > > > > > > > > > > > > > > > It caused memory-reserved region "0x0-0x1fffff" without valid node id > > > > > > > > > > > in "memblock_get_region_node" from "memmap_init_reserved_pages", lead to > > > > > > > > > > > "reserve_bootmem_region-> init_reserved_page -> early_pfn_to_nid()" > > > > > > > > > > > accessing "node_data" out of bound. > > > > > > > > > > > > > > > > > > > > > > To fix this bug, we should remove unnecessary memory block reservation. > > > > > > > > > > > > > > > > > > > > > > +. no need to reserve 0x0-0x1fffff below kernel loading address, since > > > > > > > > > > > it is not registered by "memblock_add_node" > > > > > > > > > > > > > > > > > > > > > > +. no need to reserve 0x0-0xfff for exception handling if it is not > > > > > > > > > > > registered by "memblock_add" either. > > > > > > > > > > > > > > > > > > > > > > Fixes: commit 61167ad5fecd("mm: pass nid to reserve_bootmem_region()) > > > > > > > > > > > Signed-off-by: Huang Pei > > > > > > > > > > > --- > > > > > > > > > > > arch/mips/kernel/traps.c | 3 ++- > > > > > > > > > > > arch/mips/loongson64/numa.c | 2 -- > > > > > > > > > > > 2 files changed, 2 insertions(+), 3 deletions(-) > > > > > > > > > > > > > > > > > > > > > > diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c > > > > > > > > > > > index 246c6a6b0261..9b632b4c10c3 100644 > > > > > > > > > > > --- a/arch/mips/kernel/traps.c > > > > > > > > > > > +++ b/arch/mips/kernel/traps.c > > > > > > > > > > > @@ -2007,7 +2007,8 @@ unsigned long vi_handlers[64]; > > > > > > > > > > > void reserve_exception_space(phys_addr_t addr, unsigned long size) > > > > > > > > > > > { > > > > > > > > > > > - memblock_reserve(addr, size); > > > > > > > > > > > + if(memblock_is_region_memory(addr, size)) > > > > > > > > > > > + memblock_reserve(addr, size); > > > > > > > > > > > } > > > > > > > > > > > void __init *set_except_vector(int n, void *addr) > > > > > > > > > > > diff --git a/arch/mips/loongson64/numa.c b/arch/mips/loongson64/numa.c > > > > > > > > > > > index 8f61e93c0c5b..0f516dde81da 100644 > > > > > > > > > > > --- a/arch/mips/loongson64/numa.c > > > > > > > > > > > +++ b/arch/mips/loongson64/numa.c > > > > > > > > > > > @@ -130,8 +130,6 @@ static void __init node_mem_init(unsigned int node) > > > > > > > > > > > memblock_reserve((node_addrspace_offset | 0xfe000000), > > > > > > > > > > > 32 << 20); > > > > > > > > > > > - /* Reserve pfn range 0~node[0]->node_start_pfn */ > > > > > > > > > > > - memblock_reserve(0, PAGE_SIZE * start_pfn); > > > > > > > > > > > } > > > > > > > > > > > } > > > > > > > > > > -- > > > > > > > > > > --- > > > > > > > > > > Jiaxun Yang > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Sincerely yours, > > > > > > > > > Mike. > > > > > > > > > > -- > > > Sincerely yours, > > > Mike. > > > > -- > Sincerely yours, > Mike.