From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 229A2C3ABAA for ; Mon, 5 May 2025 13:25:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 14EA26B0085; Mon, 5 May 2025 09:25:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D6C66B0089; Mon, 5 May 2025 09:25:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE1786B008A; Mon, 5 May 2025 09:25:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CE5F86B0085 for ; Mon, 5 May 2025 09:25:01 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D231980462 for ; Mon, 5 May 2025 13:25:02 +0000 (UTC) X-FDA: 83408924844.26.BD89891 Received: from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31]) by imf17.hostedemail.com (Postfix) with ESMTP id 0343F40009 for ; Mon, 5 May 2025 13:25:00 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Uh4QryBg; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf17.hostedemail.com: domain of rppt@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746451501; a=rsa-sha256; cv=none; b=YswQsuMpsR1wEc8hhX0cOj2Bq+m4VJnbj0OYB+oev+P5Nayub1C4Riqcao2y826Mppqpok yJISYYAXpnyxBtBz0kaQVssKOBZAQm8SeRqRlJyzjfIQz4ow5P2tgdh6kxRT/WR4K5ea65 NEWhjrhoNgzTHAK883xl4Jo6Ou4cDRo= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Uh4QryBg; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf17.hostedemail.com: domain of rppt@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746451501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AfQLLrJyLYjLimc3Y70sthhLQjfL2Rn+I+KkBhBSOjE=; b=ZV/q6UmfK+3J9ga9jtB4xd1DdCburavJjjT4gon+i+KdUe761SeySOpBf//FGE9pyCbgh4 PT50GMwuOx0SCxcUEFJMfyvmimmlQqmcqD7BRXZQ9wADdn69uJ3GvKvNAp2srlzqTRoUae 0NSKsJJE5TW6ooZ5Kc7J5Ym5KGn6t3g= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id F37B843979; Mon, 5 May 2025 13:24:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6CB47C4CEE4; Mon, 5 May 2025 13:24:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1746451499; bh=zlAQRQheJk2TEOPVNZOX1pRLQFKGRuJfcSAymuaD38o=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=Uh4QryBgTMLKOFr1BKuTWSCYKmofogpzIoMKuQVI0J2dd9G6W3DXHycjuUcygDdF/ M0qom0HJSu5hllsvCDs/+3at4xwbvuwCu91SLt/gp5fx003vJoUAwLMOYco8lI0N/g AJBIHdNxWA2lhWvk9w/p+iXqhxcPCkqMR0PjUyf819uKnL8UkR7mbn+D8wLwOyNfAr K8E0s+jlvoS14J9tV6czMW7fxN8rrCZNJPm5W8MJqMcJnTl5TpCs58nhLX2YBRY+H0 v8kU1rQ4gWKYB5bP9z/J8rF1EkUxR+ptGGq3Serd2qtq3FRk7n1rCDAzuUQNGP6GBo poRxhx9NIC4lg== Date: Mon, 5 May 2025 16:24:50 +0300 From: Mike Rapoport To: David Hildenbrand Cc: Oscar Salvador , Donet Tom , Zi Yan , Greg Kroah-Hartman , Andrew Morton , rafael@kernel.org, Danilo Krummrich , Ritesh Harjani , Jonathan Cameron , Alison Schofield , Yury Norov , Dave Jiang , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v3 1/3] driver/base: Optimize memory block registration to reduce boot time Message-ID: References: <188fbfba-afb4-4db7-bbba-7689a96be931@redhat.com> <74c500dd-8d1c-4177-96c7-ddd51ca77306@redhat.com> <8180a50d-eebe-4f9b-9ce8-d886654a992d@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8180a50d-eebe-4f9b-9ce8-d886654a992d@redhat.com> X-Rspam-User: X-Rspamd-Queue-Id: 0343F40009 X-Rspamd-Server: rspam04 X-Stat-Signature: f5wdpjuzwpzhkmz1hruer3iayfwu895g X-HE-Tag: 1746451500-905554 X-HE-Meta: U2FsdGVkX18q3U0ofHYhb9mYH2FE7XrboR03kDX4TbqbQnjco3hBGFGtPy3nv5gKBRtzZprjznP7Svz7pK66CbTAG9eA2BelFSFA6yxV4MNlknGaA2Q9p8UjgVylBF8+lwxd4GHR96QBl10ngezujmx37ooMq+XtVT4XYRTM2Q/XLX8/YGndApu5BdhDtO8YGFB72rxuaKebx2lEyNZ4JOBGEc8i/Jxv3dYg965WwF/rJoQ+wdPCOMT0qxQyf3Yqpg0xM3ti7rhWhdlBhh0/RqCAhRY1N1zZqmjlNiSBa27KQGdx5J2IKI+xCZGc2axvHsEY6ARK1EDiLhJFRo2e3qBMXVTviO9TZArFYY4VmC1HYZTQy8kzXLvuQ6zIWKAR0Lht40qpeW8ntb0Dyob0MfbRgPy5q4GCDBE4FIcX4aueMrWC0a7Te0KA+itffMNT9zRv4HYGYrCktPioJTIqAXaDnXGFTLKjIBKIo1Crd9aH1TvLzfZfDfVjdRVfrJTnOOkuLQyHsEPCeNEkEdWSQOnQz6vpnoiOzNLY3vIJsOu8zVSxZchKiGr5fU2DjZj6gUcg1Ww3dKYpbtx4VM32DRQUiT7cZeWsag3grwW//bPhoqktixqQBqTd1s213Ct5TH6NheVbC16XG3Nkd32tSqqaLugXOtpOp9RYwV/TvltQIibPyOMEO35ACxBDEf5Jz/DUQ18jmeLXJSk1tDvr3ku0MZmy/lYBBzf2U3LIoo1/3X78S7KGyNJpux3a1c7Ud+goMGBSjcz7E9nhpopYCrjxwJEyfcGBZhTqPxifNiZA7+IBgonqdFzsHAXd1vOzM8Z5DJulovTzqlpxmwI/aPiKAkXi4wV9pM5d6R8etSNn4t+xdJ5GdVJFwxye/npCtZ/KLpgg4DH9Z9cNWqf9jYdBewbv6+RTo8u0zTkOHlfKdAreW6TUTLW+V/DDC7LiMuwILwueFvzNb47sgV8 8VxRzh5Z SqUye6bT4clDvnfL11BKhJ61n8qLPRV20v8rVGJ5GRcpNU547GB9ObKI/h2wpL0c8XhkvyQA0D6nDonub004+phN7z1ndDEpmVOjIobprDk70qFer4KkaCnyVLpwXsGA+Nfnbir2+uob/DquGOCU3ETq4bALI3kP59Oq0q/DzAznwJHMJV22lZM/Yiq6xeqINGFFHnYjXzm6HnP2I5LDa5M1zPBS/Y1Un2Z9Q9NubhGVDTan/jcrXw6GBOM9T/DniSqjrmKlCDtWiatj4kpHiq56KBwvC67M8xDmL0WPQhZjPxgjhJfZCMvDAtWu6rIWdT/hrWKWEMhqE/R8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, May 05, 2025 at 10:18:43AM +0200, David Hildenbrand wrote: > On 05.05.25 09:53, Mike Rapoport wrote: > > On Mon, May 05, 2025 at 09:38:43AM +0200, David Hildenbrand wrote: > > > On 05.05.25 09:28, Oscar Salvador wrote: > > > > On Mon, May 05, 2025 at 09:16:48AM +0200, David Hildenbrand wrote: > > > > > memory hotplug code never calls register_one_node(), unless I am missing > > > > > something. > > > > > > > > > > During add_memory_resource(), we call __try_online_node(nid, false), meaning > > > > > we skip register_one_node(). > > > > > > > > > > The only caller of __try_online_node(nid, true) is try_online_node(), called > > > > > from CPU hotplug code, and I *guess* that is not required. > > > > > > > > Well, I guess this is because we need to link the cpus to the node. > > > > register_one_node() has two jobs: 1) register cpus belonging to the node > > > > and 2) register memory-blocks belonging to the node (if any). > > > > > > Ah, via __register_one_node() ... > > > > > > I would assume that an offline node > > > > > > (1) has no memory > > > (2) has no CPUs > > > > > > When we *hotplug* either memory or CPUs, and we first online the node, there > > > is nothing to register. Because if there would be something, the node would > > > already be online. > > > > > > In particular, try_offline_node() will only offline a node if > > > > > > (A) No present pages: No pages are spanned anymore. This includes > > > offline memory blocks. > > > (B) No present CPUs. > > > > > > But maybe there is some case that I am missing ... > > > > I actually hoped you and Oscar know how that stuff works :) > > Well, I know how the memory side works, but the CPU side is giving me a hard > time :) > > > > > I tried to figure what is going on there and it all looks really convoluted. > > Jap ... > > > > > So, on boot we have > > cpu_up() -> > > try_online_node() -> > > bails out because all nodes are online (at least on > > x86 AFAIU, see 1ca75fa7f19d ("arch/x86/mm/numa: Do > > not initialize nodes twice")) > > node_dev_init()i -> > > register_one_node() -> > > this one can use __register_one_node() and loop > > over memblock regions. > > > > And for the hotplug/unplug path, it seems that > > register_memory_blocks_under_node(MEMINIT_EARLY) is superfluous, because if > > a node had memory it wouldn't get offlined, and if we are hotplugging an > > node with memory and cpus, memory hotplug anyway calls > > register_memory_blocks_under_node_hotplug(). > > > > So, IMHO, register_one_node() should not call > > register_memory_blocks_under_node() at all, but again, I might have missed > > something :) > > Hm, but someone has to create these links for the memory blocks. My understanding that the links for the memory blocks during hotplug are created in add_memory_resource() register_memory_blocks_under_node() So register_one_node() only calls register_memory_blocks_under_node() when there are no actual memory resources under that node, isn't it? Then we can drop the call to register_memory_blocks_under_node() from register_one_node() and add creation of memory blocks to node_dev_init(), i.e. node_dev_init() for_each_node(nid) __register_one_node(nid) for_each_mem_region() /* create memory block if node matches */ > It's all very messy :( It is :( > -- > Cheers, > > David / dhildenb > -- Sincerely yours, Mike.