From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67CE5C3ABBC for ; Fri, 9 May 2025 15:41:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F9966B0089; Fri, 9 May 2025 11:41:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 681886B008A; Fri, 9 May 2025 11:41:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F98D6B0095; Fri, 9 May 2025 11:41:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2CB026B0089 for ; Fri, 9 May 2025 11:41:00 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 34E76160332 for ; Fri, 9 May 2025 15:41:01 +0000 (UTC) X-FDA: 83423782722.29.95F331B Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf07.hostedemail.com (Postfix) with ESMTP id 9BEA44000F for ; Fri, 9 May 2025 15:40:58 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=hB5hJo6F; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf07.hostedemail.com: domain of donettom@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=donettom@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746805258; a=rsa-sha256; cv=none; b=p1n9eN7MvQZoXh/kbAW7c2xJboNVozpicVFlIMLEmOCO1yKmNM3q9Gs+v2LOUsOfsldYqM sptWfqZK3GLGllm7C1aVgLsHccZsckqKuBmE7xxhDnyWmqyjqDyRFzwGKwymQwar6jgmcQ mXjkOwKOn09IPozzDoaMeclvd2cXQ9w= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=hB5hJo6F; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf07.hostedemail.com: domain of donettom@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=donettom@linux.ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746805258; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pmSKL4D4kH+Gt6bFEcGGXb0xIWjN2YDtb8WLtaVT54o=; b=7fgZojJAd2TQcocppi+amMTAzqRCXoZQLFZFcqAX/r0MYvoBeP0YZdMMtW4k3m5bLDVxFS 6X0noV/3+oDvjEio0awV9yazTAx1JxpZTVnnBvLAWmvKRWsKbO1lOh7qanp72fK/unbnst Xnm1XWmdGqm8bMYzO+KIHHQbUo1XJgk= Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 549DUACr029559; Fri, 9 May 2025 15:40:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=pmSKL4 D4kH+Gt6bFEcGGXb0xIWjN2YDtb8WLtaVT54o=; b=hB5hJo6F5cVfibmDNrlQI6 TQ7WDbnoenMOKEEyxn/Q+QR4whVr7zkfPKYN7Wk5UYKfyk8SNROE9hrUiJDINcXW a0kow/XgfLkl0GFUpNt2s8Yj0fRdJ6gg2A8q8mycT8afMSdj/mL9hJ4uOGsD0TKX LvqySclbIPYmRi3iinBOcgwB6a16Id+5trrMExNF1umjjNxOd586OFr2wE7Y45Gg LZAcL0eseuQ7V0SpWgZr7lA9qBhxyhK3u2MFw7LXTmBTogobQ9QcS9pm4PtSKDN/ qtS93tkUU2vpjWgrlS30iVrbnT4QLHI0f4Aj/vZ351o5kGpRVZUdVkaq1scpBxUw == Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 46h6k0mak9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 09 May 2025 15:40:43 +0000 (GMT) Received: from m0353725.ppops.net (m0353725.ppops.net [127.0.0.1]) by pps.reinject (8.18.0.8/8.18.0.8) with ESMTP id 549FYoIJ009045; Fri, 9 May 2025 15:40:43 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 46h6k0mak4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 09 May 2025 15:40:43 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 549E1SUt014583; Fri, 9 May 2025 15:40:42 GMT Received: from smtprelay07.wdc07v.mail.ibm.com ([172.16.1.74]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 46dypm3wkb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 09 May 2025 15:40:42 +0000 Received: from smtpav05.wdc07v.mail.ibm.com (smtpav05.wdc07v.mail.ibm.com [10.39.53.232]) by smtprelay07.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 549FefQS15598152 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 9 May 2025 15:40:41 GMT Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 99EBB58043; Fri, 9 May 2025 15:40:41 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DAB9258053; Fri, 9 May 2025 15:40:36 +0000 (GMT) Received: from [9.124.209.250] (unknown [9.124.209.250]) by smtpav05.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 9 May 2025 15:40:36 +0000 (GMT) Message-ID: Date: Fri, 9 May 2025 21:10:34 +0530 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 1/3] driver/base: Optimize memory block registration to reduce boot time To: Mike Rapoport , Andrew Morton Cc: Oscar Salvador , Zi Yan , Greg Kroah-Hartman , rafael@kernel.org, Danilo Krummrich , Ritesh Harjani , Jonathan Cameron , Alison Schofield , Yury Norov , Dave Jiang , linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <188fbfba-afb4-4db7-bbba-7689a96be931@redhat.com> <74c500dd-8d1c-4177-96c7-ddd51ca77306@redhat.com> <8180a50d-eebe-4f9b-9ce8-d886654a992d@redhat.com> <78bc6a1b-164e-4925-a624-a271a4499364@redhat.com> Content-Language: en-US From: Donet Tom In-Reply-To: <78bc6a1b-164e-4925-a624-a271a4499364@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: uZaB9WtL4eQ4mY4iz0NpmkauIXeMA1hl X-Proofpoint-ORIG-GUID: JBUuPV6qFmy1fGheTknIVlXLho2ryNpA X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNTA5MDE1NCBTYWx0ZWRfX9xQDvNYZD6SJ v7hiPtAHyzz301hFpLNdoQ1AXXZAvniOtN8bpFbXhWZsbMuu7EVhTBuHKnbBoGEnh5vIydBD/Z/ kJttl7zjbil7T1IT47kktq3J48jyu+U5iBWf8V06wMoORphcen00URkz1B2cGWJTn6Ng1BEvYsS NFUloD8jbQzDV+F1i6q0q80mfCYO/LyAwBvux6M18wyTZ6IqwxXiVeoB8wgtwcf034N4mGgW16R 33zL13yTDg6dN2BTcXnAviXNQm+YZokSucIJy4/FRVaJoqDnvOywCVeaNBkXRcYdHSzDV6m4ryh zzU74xeszmaYCwqPCEbghiQQdfsGEfbqBqYZh9FDjSYmL2e7dd4XP+q2GnZNFxKqCSO+GE9wF06 yTBCxvA0DMaC7PwsO74O6/D2tD3+HvRGulfL+IrK0p5iRwU7oQ13ssOqmEQSIWHldZWLz9GF X-Authority-Analysis: v=2.4 cv=OcCYDgTY c=1 sm=1 tr=0 ts=681e21fb cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=IkcTkHD0fZMA:10 a=dt9VzEwgFbYA:10 a=erVvInnpyg78pcfNAL4A:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.736,FMLib:17.12.80.40 definitions=2025-05-09_06,2025-05-09_01,2025-02-21_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 lowpriorityscore=0 phishscore=0 spamscore=0 bulkscore=0 impostorscore=0 malwarescore=0 clxscore=1015 mlxscore=0 priorityscore=1501 adultscore=0 mlxlogscore=792 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2504070000 definitions=main-2505090154 X-Rspamd-Queue-Id: 9BEA44000F X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: hnxmkkj7n7ne5e34yznx5z4g7cbhbu9t X-HE-Tag: 1746805258-979307 X-HE-Meta: U2FsdGVkX1/NUOphGjyNQMKKQaoirchcD3pdxy9UY74PbkWfpEDNKHcYdul2/lNvf2gPST+O6cpmsZ1eQuhoD7Q2oRCh+k6Pbw5I3G46Pltc99+EgEK0vnw8d6lUADPrYxowWJYHVZiQNz8IuRJZ0Zap9xMYmxN+u29NGfTkt36DkB7wAe/EX0r9AVHbJdJ4JHeoTpB5GZApm3Pwp63QyaHHyizl6jBIvmOqVxoYI2nzxWjxNtjmzvXgppnW6fYFGx8CSI3Dn1fAHPZL3Hg03aG8RyW5RJJT6no+DWTMilcM64tq5SFXYIs1g4NxXaT72AJMJ4Ue8lf9ITDDQp1Munn0U0eIJ8q/2ckATjuysZB0Pt8hxLzzdmadYDzbQ8Gwr/wu6Q4iIm87VY4fTs0evdcnElu9/q81Y5f1N07cVUfekL6XFSLwNEfWcpTrLMyVmIZDE+0UEd35SEv6fU1g32UlAXyiTFM9X+xSKgj8rIwrzT5bMyVArb+CYDo8z7yPYl+//Aw8StuilYbyGRNYQCZw1eSLh/5Gjzz4q4h91l1itBM2ZuMHEtgH8a0fTLcRdqgJkpiypYudmP0xZVtLMFnn0zn0KO0kLDYZmiFIg7Bm96MjPrjX/vWFE7Ck9b9+uWmbDCVD3JdwZ5vG0zR44IRUKASDrCDgHtW6QdBcOH9mvAe3rsgzNC2Y6JAeXBGd4okg8KVzd2pUzn7pgzpLlx8v2/zV2YMCHSo/kaHWqXJkQlsvn7I3X9hF9H2VxANMtoeGUvyBy+4bzztyeuw+3TmvnsFvH4vZNrmklVF4Zt0TnCydWfUH775x3rBB1k5WWbLfGMXB9zgv6WHo9k7q3GFc3P5awyzhHG8Y7Kk9oLfyzInTodC80dAeeuWIT8Eze7WUMH+VUKGonv9GJdxhCiiA0otvTveBlfOsrEAhmGjtVwNKRRsBbk2ZoSroAktOCNGfsHZ+f1MsAVECNy+ axWLI6K5 BHO6Wml+RMoeZaNn+Hj7GkuHV3z5n7oFihzNJy5PrNl1Nmnqsz5IzX/NMI0SmVOs7eB8tEmjj42LrRWiVfZV1cWZYqJN/Z4KxCrKT54I2bJI5cmRkAnj7XS8ZR507ycs8Jh1JvUm82b9GesRTYTWmLhwXHwO2igeqOF1hES58GNV+ztGyxM5i7EmZvFd2ofG7Gj7cLDXyahw/xInQhRIyqp2+Kh9pxAQ7YgmFgV6bCQf4VrMiYoxigrjsv74hV7EA5FfsgvBF1zEVZPlDPfSmx67GWgeFk08dEG4uKj4OMUXgm3pQFUe75WPS9Lenifj+q0G1N/CtETn/C8gfuSsFmz7kg8lcbdjnm/uHhe0+bbsU1iSZSrN8+ZMV7NGhtxGETrupdrgOPJR23f3L6sD6Rmaxekr1rgE49TWnVfhxAjb/JRA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 5/8/25 2:48 PM, David Hildenbrand wrote: > On 05.05.25 15:24, Mike Rapoport wrote: >> On Mon, May 05, 2025 at 10:18:43AM +0200, David Hildenbrand wrote: >>> On 05.05.25 09:53, Mike Rapoport wrote: >>>> On Mon, May 05, 2025 at 09:38:43AM +0200, David Hildenbrand wrote: >>>>> On 05.05.25 09:28, Oscar Salvador wrote: >>>>>> On Mon, May 05, 2025 at 09:16:48AM +0200, David Hildenbrand wrote: >>>>>>> memory hotplug code never calls register_one_node(), unless I am >>>>>>> missing >>>>>>> something. >>>>>>> >>>>>>> During add_memory_resource(), we call __try_online_node(nid, >>>>>>> false), meaning >>>>>>> we skip register_one_node(). >>>>>>> >>>>>>> The only caller of __try_online_node(nid, true) is >>>>>>> try_online_node(), called >>>>>>> from CPU hotplug code, and I *guess* that is not required. >>>>>> >>>>>> Well, I guess this is because we need to link the cpus to the node. >>>>>> register_one_node() has two jobs: 1) register cpus belonging to >>>>>> the node >>>>>> and 2) register memory-blocks belonging to the node (if any). >>>>> >>>>> Ah, via __register_one_node() ... >>>>> >>>>> I would assume that an offline node >>>>> >>>>> (1) has no memory >>>>> (2) has no CPUs >>>>> >>>>> When we *hotplug* either memory or CPUs, and we first online the >>>>> node, there >>>>> is nothing to register. Because if there would be something, the >>>>> node would >>>>> already be online. >>>>> >>>>> In particular, try_offline_node() will only offline a node if >>>>> >>>>> (A) No present pages: No pages are spanned anymore. This includes >>>>>       offline memory blocks. >>>>> (B) No present CPUs. >>>>> >>>>> But maybe there is some case that I am missing ... >>>> >>>> I actually hoped you and Oscar know how that stuff works :) >>> >>> Well, I know how the memory side works, but the CPU side is giving >>> me a hard >>> time :) >>> >>>> >>>> I tried to figure what is going on there and it all looks really >>>> convoluted. >>> >>> Jap ... >>> >>>> >>>> So, on boot we have >>>>     cpu_up() -> >>>>         try_online_node() -> >>>>                bails out because all nodes are online (at least on >>>>             x86 AFAIU, see 1ca75fa7f19d ("arch/x86/mm/numa: Do >>>>                           not initialize nodes twice")) >>>>     node_dev_init()i -> >>>>         register_one_node() -> >>>>             this one can use __register_one_node() and loop >>>>             over memblock regions. >>>> >>>> And for the hotplug/unplug path, it seems that >>>> register_memory_blocks_under_node(MEMINIT_EARLY) is superfluous, >>>> because if >>>> a node had memory it wouldn't get offlined, and if we are >>>> hotplugging an >>>> node with memory and cpus, memory hotplug anyway calls >>>> register_memory_blocks_under_node_hotplug(). >>>> >>>> So, IMHO, register_one_node() should not call >>>> register_memory_blocks_under_node() at all, but again, I might have >>>> missed >>>> something :) >>> >>> Hm, but someone has to create these links for the memory blocks. >> >> My understanding that the links for the memory blocks during hotplug >> are created in >> >> add_memory_resource() >>    register_memory_blocks_under_node() > > Yes, during hotplug it's exactly that, after registering the node + > setting it online. > >> >> So register_one_node() only calls register_memory_blocks_under_node() >> when >> there are no actual memory resources under that node, isn't it? > > Except in early boot. That's why register_one_node() has: > >     register_memory_blocks_under_node(nid, start_pfn, end_pfn, >                       MEMINIT_EARLY); > >                         ^ early :) > And that is triggered by > >     node_dev_init()->register_one_node() > > >> >> Then we can drop the call to register_memory_blocks_under_node() from >> register_one_node() and add creation of memory blocks to >> node_dev_init(), >> i.e. >> >> node_dev_init() >>    for_each_node(nid) >>      __register_one_node(nid) >>        for_each_mem_region() >>          /* create memory block if node matches */ > > Yes exactly, that makes sense. Hi Andrew and Mike Based on the discussion so far, it is clear that the patch will work in all cases, including when CONFIG_ARCH_KEEP_MEMBLOCK  is disabled. Just checking — would you prefer to take this version, or should I send a v4? Thanks Donet