From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCCC2C3ABAA for ; Mon, 5 May 2025 13:07:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CFA16B008A; Mon, 5 May 2025 09:07:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 680176B008C; Mon, 5 May 2025 09:07:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F8B46B0095; Mon, 5 May 2025 09:07:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2D5386B008A for ; Mon, 5 May 2025 09:07:28 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B336D80438 for ; Mon, 5 May 2025 13:07:28 +0000 (UTC) X-FDA: 83408880576.13.BDCE75E Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf24.hostedemail.com (Postfix) with ESMTP id 768C0180013 for ; Mon, 5 May 2025 13:07:26 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=UXbmpPeU; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=45CPe9Wg; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=UXbmpPeU; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=45CPe9Wg; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf24.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.130 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746450446; a=rsa-sha256; cv=none; b=kUwzoFPP3fUykygm4tfBcBmd5mY4LYuU2b3OlV2spk3VlOXVDYfLokoSCufPYHNlq9jS+i Sfh2anvsWQ4LqKb0UdLq/tWwSN2EOslJruxK1Hy69UkvFcxSL8P5ILPKba2RsDfeBZ9DbW FVIpeHfBQQXfGV9whLfRFD6q02h8TSw= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=UXbmpPeU; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=45CPe9Wg; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=UXbmpPeU; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=45CPe9Wg; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf24.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.130 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746450446; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n7bM99N3Kh6W6XwDJeCpv3UvEQyAWW+ApgUPey55Uew=; b=wjKFU602miZrsyCS+k9SuYHDXro4FpqKAGK0axj8HNWDEQdjthS76iHsCsOudn/DnGM7Gb 60YAXrAiD1NjhidK3YxXPqEuMhb8jwfrvP64j4PpMNAWa2DYhttatS3PNHLL57BpvB78h+ PrqgZ9tzbdfMsCAnpOPIY269L2f0qeM= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 9D297211EE; Mon, 5 May 2025 13:07:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1746450444; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=n7bM99N3Kh6W6XwDJeCpv3UvEQyAWW+ApgUPey55Uew=; b=UXbmpPeUdI6RZBQM7QlcUE1bplF6Ccf3krG6pfyeNq5NkeGFzmium+TgD04qDiAUVV0AKh pXgIWwpXoAe8k/GIZxi3pOZRN+OUM4XLo0+jWCQ15Rvx46scwQS1YNAkeqlDxavHegsPvl T5h6gJ68crqGkBQGMGJVZLT45jQe3pw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1746450444; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=n7bM99N3Kh6W6XwDJeCpv3UvEQyAWW+ApgUPey55Uew=; b=45CPe9WguxqpWyWbrjzPN382WBQ0Baf6JOAShn1QfglbH34CHmrg997AoKo6yMZ4h50gYD gXhG+XoNL0sIL2DA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1746450444; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=n7bM99N3Kh6W6XwDJeCpv3UvEQyAWW+ApgUPey55Uew=; b=UXbmpPeUdI6RZBQM7QlcUE1bplF6Ccf3krG6pfyeNq5NkeGFzmium+TgD04qDiAUVV0AKh pXgIWwpXoAe8k/GIZxi3pOZRN+OUM4XLo0+jWCQ15Rvx46scwQS1YNAkeqlDxavHegsPvl T5h6gJ68crqGkBQGMGJVZLT45jQe3pw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1746450444; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=n7bM99N3Kh6W6XwDJeCpv3UvEQyAWW+ApgUPey55Uew=; b=45CPe9WguxqpWyWbrjzPN382WBQ0Baf6JOAShn1QfglbH34CHmrg997AoKo6yMZ4h50gYD gXhG+XoNL0sIL2DA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id CD65F1372E; Mon, 5 May 2025 13:07:23 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id JPRMLQu4GGjlLQAAD6G6ig (envelope-from ); Mon, 05 May 2025 13:07:23 +0000 Date: Mon, 5 May 2025 15:07:17 +0200 From: Oscar Salvador To: David Hildenbrand Cc: Donet Tom , Mike Rapoport , Zi Yan , Greg Kroah-Hartman , Andrew Morton , rafael@kernel.org, Danilo Krummrich , Ritesh Harjani , Jonathan Cameron , Alison Schofield , Yury Norov , Dave Jiang , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v3 1/3] driver/base: Optimize memory block registration to reduce boot time Message-ID: References: <188fbfba-afb4-4db7-bbba-7689a96be931@redhat.com> <74c500dd-8d1c-4177-96c7-ddd51ca77306@redhat.com> <0e568e33-34fa-40f6-a20d-ebf653de123d@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Action: no action X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 768C0180013 X-Stat-Signature: 6t16ea3kj3jwp5fdkb631ai5dnfonoi5 X-Rspam-User: X-HE-Tag: 1746450446-889241 X-HE-Meta: U2FsdGVkX1+CbPV7Zrwy8ZTU90j1+WnZ3xV7hFvlDsRnCZiCiAb7wq7FSRoerAH8RPUc93VyareI7wzWpm4VOl1ISJjMkbHwj83iAUqofPH6JKXrFjCc0kDhul1f3w52P95GGOmJSI9iZrv7HidQrr6WyjRbCBzw90PDoUWb3CjOGyRWeIEpro3Mf20qOE8sATFnTTLfdEsm0dd8ysill+LiKnwpCRO2fggIup2q1QVTyADWLud/Sa9IEtH5y0fmQYQYKuio8eQfDtUeYeuA2aUgpdZqfme4IsGHidPn/V3Swe+pvLlkS73XkqUSd1kY/+rMvLfR9rAnK5V5JFQOef83UqLlrTTfhrOl8wQafan0QitjuDKT+EF8Ru+tmVsIND8R+1UP/jF4BgxFzxGkAIMVeVZKKkzJg2akiADHUwxSZa0V4lQ6Cg38MVJ8mjAsFk5GYn8rPP5Gaj2hXhkcHwisWIhE0wOrGkxf/4OjDu18OtYNZ2GPtDN9qKAkCQX2MOlQ+adGIapG5CKHf7VcitDBiWWtC1J3qSOTP1pBrx5Wb79CiD24J7CyJYByA61OYdMsMh9h7HHsz7fnrJZ84xdIHYU3jiIepeVJb2vsnLONzbPF2i5zHWRM0YKqKZ9ZAfXHEACl0g1cqqR5yOJq+BvpwZ5J7nouhhD1LuxtCb196fBce+GJjIkiXKELDDJj4UbaQc+U4QYV9vojoxTfqHd8E9O64X/40sG7WKchawaYC0d+NZBnONnwNjpde1uiQhwE/MuQ24vVfsp1TvBJUb72RwQXuJ7vZNyV+Qq50mET1K5cQKLGYSqc11+xUWIQ5GJ1AbrKkXwBOjBnXP/Qtv/O6948QVhMsZMo8xZS6DNF6ZxWY6YqEfiol5x/QW8w7U/pLtIjmAuFbS0OnMSZ9K75odnFhidu6Sznn23BuV2dijbNGJDocAZV9X7GV6x6gqwgEEugmCWJkinqIFj kAd751Mz Rds/aZKkCIratpPOOBJwsp7/ih8HJD71NT5M1iyUs5/Wqc6p7f6bQXKWhtn8ln4vPYaqI2O61Rtz8/hIfBui6A1A/vPRfPLWUSjbEjh+JEgqTqy1AnnMAOtn+H/N4OHs1RL25hX93/7mczZfNXREN7iYwAnVpeTKNFmnU8QPyy28Uc+gvcPhbHKCUciIysT7F9pzRp7LULCGeNBztSkDP3/wMjp+DEKYVTVFDnZIaUr6kJl/7aY8hFbe0sVGT2j9MFahr6wzCoRJNka5sqveH7jcYFwdc9OvVY6jsWv4XOcsObfBkahqX1v30tFP/fjM92EUekbuQx4/yo6U= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, May 05, 2025 at 12:36:21PM +0200, David Hildenbrand wrote: > My thinking was, whether we can simply move the register_cpu_under_node() > after the try_online_node(). See below regarding early. > > And then, remove the !node_online check from register_cpu_under_node(). > > But it's all complicated, because for memory, we link a memory block to the > node (+set the node online) when it gets added, not when it gets onlined. > > For CPUs, we seem to be creating the link + set the node online when the CPU > gets onlined. Yes, that is one think we need to align on. So, add_memory_resource(), which is part of the hot-add stage, will mark the node online if it needs to. I guess that this is done this way because further down the road we rely on the node to be online for some reason (e.g: online_memory_block() being called from add_memory_resource()). Although from the conceptual point of view, it does not make sense, does it? I mean, if a node does not have any resources onlined, why should it be onlined? But let us assume that that is the way to go, then we could in fact tweak the cpu-hotplug code to do the same, and online the node whenever a cpu gets hot-added, and not really waiting for it to be onlined. I can do some experiments about it and see how it turns out. > > The first time we hotplug a cpu to the node, note that > > register_cpu()->register_cpu_under_node() will bail out as node is still > > offline, so only cpu's sysfs will be created but they will not be linked > > to the node. > > Later, online_store()->...->cpu_subsys_online()->..->cpu_up() will take> > care of 1) onlining the node and 2) register the cpu to the node (so, > > link the sysfs). > > > And only if it actually gets onlined I assume. Yes, we only register it if we managed to online the node. > > I think that ideally, we should only be calling register_cpu_under_node() > > from register_cpu(), but we have this kinda of (sort of weird?) relation > > that even if we hotplug the cpu, but we do not online it, the numa node > > will remain online, and so we cannot do the linking part (cpu <-> node), > > so we could not really only have register_cpu_under_node() in > > register_cpu(), which is the hot-add part, but we also need it in the > > cpu_up()->try_online_node() which is the online part. > > Maybe one could handle CPUs similar to how we handle it with memory: node > gets onlined + link created as soon as we add the CPU, not when we online > it. Yes, indeed, as I stated above I think this is the way to go, to have it paired. > But likely there is a reason why we do it like that today ... I cannot think of any (Tm), but it is a possibility. I will take a break from hugetlb stuff and play a little bit with this. -- Oscar Salvador SUSE Labs