From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 373ACC021B2 for ; Sun, 23 Feb 2025 02:46:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8ED776B007B; Sat, 22 Feb 2025 21:46:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 89DBF6B0082; Sat, 22 Feb 2025 21:46:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 78BD96B0083; Sat, 22 Feb 2025 21:46:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5CCAE6B007B for ; Sat, 22 Feb 2025 21:46:35 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C88E8C1EB4 for ; Sun, 23 Feb 2025 02:46:34 +0000 (UTC) X-FDA: 83149671108.29.3F7D59C Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf27.hostedemail.com (Postfix) with ESMTP id 0C40B40008 for ; Sun, 23 Feb 2025 02:46:32 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=xYX9Z940; dmarc=none; spf=pass (imf27.hostedemail.com: domain of akpm@linux-foundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740278793; a=rsa-sha256; cv=none; b=7PsqmGBhdiPlm0qizRlTpKzSHQ96YfFA/SMApL6r4h1NWcK2e966DR76TXof381jMjA39i E6om9pnuBzfIxQphAjVN4vfJ9X3E3OfVrUrpR7rKBUWw2lZs9EJP/aUyuq9HncDVxD8+CY LGVs1gXt2zAMUyil0u3W7MN29oY/ZEk= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=xYX9Z940; dmarc=none; spf=pass (imf27.hostedemail.com: domain of akpm@linux-foundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740278793; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SItkelC+slVNz50hGA0pTijkpIuBAH+rMhiAGIWwdsY=; b=4tTGw85kbke75K8wAUpjsb9uDrgvfol+qtohSqSzQoLsLvGWi5EtQFRHfSf5aIMRUdNFaW kJmjGhOlZ92uRlyAuCVucMPqAIvDKpDtIk0Mxhsr9y9879NUpS1ICveXOi4T7EAoa/Uws+ Q+fNicPfUK2DGFEBJpxVYAwKwW0YEEs= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id CA9EE5C61B9; Sun, 23 Feb 2025 02:45:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 22F49C4CED1; Sun, 23 Feb 2025 02:46:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1740278791; bh=dhYI81vtV3obBhOyd6vtWZA2PuYEd7yu7su4Ec03D9Y=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=xYX9Z940qj9qMlHFAeliNqDDjM6RMZ1xSMUkk8IKicYPc2dqCJMOkZoj1GRFQGNsC pRhoTTAlfDqhTITuQ1z5B0hn6FNgDQKP+FGR7hWVCjOzXxNenojgwlU5y0+v0q6fg0 4X5q7lbo6B4Hxq2jEL/IK0PHv4/w6yaQbTRgHsKk= Date: Sat, 22 Feb 2025 18:46:30 -0800 From: Andrew Morton To: Thomas Prescher Cc: "willy@infradead.org" , "linux-mm@kvack.org" , "corbet@lwn.net" , "muchun.song@linux.dev" , "linux-doc@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH 1/2] mm: hugetlb: add hugetlb_alloc_threads cmdline option Message-Id: <20250222184630.1f25865325eced9b0f37eb85@linux-foundation.org> In-Reply-To: References: <20250221-hugepage-parameter-v1-0-fa49a77c87c8@cyberus-technology.de> <20250221-hugepage-parameter-v1-1-fa49a77c87c8@cyberus-technology.de> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Queue-Id: 0C40B40008 X-Rspamd-Server: rspam12 X-Stat-Signature: ew1m9r6baiuixqipdng9gnceros88geh X-HE-Tag: 1740278792-65699 X-HE-Meta: U2FsdGVkX18Dhrg+AGUtgVXvYCB1A9scSlNqRBpXeYJnHFfgIaNvdn8zI12grNSuiMhE8+tlR6Xm/qim5ID4gr/zSLk/lXnRvr4gJpOoFQuXfk38Yxlvjdp8PAELjwhMB4Id5IX2qmpglp50urqs3Kx5I9apeX8X+OcSziEHQMOniZ31VDfZpKLzzkN1VvYICtzXMalnnyfhepJEQ4atHyIf8cZn9Y59cl1lZ0LlLrRVE8WQTTu932s4tCMNcn5/131DGz5FGmgOy8Blv2Qhpao9wxRc/TBIUZ9yFfTHVuPNXiG7+hG4HFmWmKwmnNwiLYT955FR/SBkiE/mS9Vmxs5lJwDAG9cUONfcAmdnzMGO4HKNpjNc78NFKUBpOYuulvoteEkpRuz4wSMbD73Ree+YcOlx2Vh6+yDcLRkOfpICT/LG3+B91o9MuCebz2gczshN/FXibe635/j/YRyMUvjCGRmlQfB4LY01tVgsICMKxk76g5G36GIIrgir2iu+0IDbSNgp3lcPDSywh5REJYpJ/dZ+K2PGHej35XDOQpRNx0+ISv89aKsxceHh53bVOLwqPmUMxChKIy0vW0hqeAulSy5jOQtrJqmNcVE+5t+RjQWOcu/+atowvPzTTOlJ7mO8r5iw2ixoJYpctkj8xiLdYNUIhE19ypAgWZ3wLAEvdzT0Lw7J2E4LSB44ZilP6OAufWQ8ImUYMYk0gDawt2C8zpJvLAtaNxj9l+Rdy7e5pB6ALBvQ73AeSW+bVrvaQ+dzQAzKO1ihSMHgoQ7ipIo2MVDq68XfHth+KLfm+hhEQB1ONBqy8nCTEJeDRg9fajiL/phbt/5QMZnYdZqDHjL3xSe0hxEjf8IqDscQJ2SifVqWCbdvSpwY7lqO9vLiUrAFS+mG6D6wnBcsmP80ttX53TsC2LbNibPbwRXrN1FrG5iL/l3RRi0c4PcUPz6mg0qk8ZHI1qbZOfwqfLU PzmN0kh2 GaMQgA7KHTCAQmABFLid5j4oUHdEHfzGFhEZtJxzDwBBf5wmnAEfbRbTJmZY+PlMInZ83/bFWpk9Lu7ICYooE2ANq5U8dZ/4G2z0f60iaUMLyLsZvlDhIy/ooaD02Llh/2n6lDQs9rUjcCUf/jtzMq5NZk6Aa4mOJNEIVr9SuWD0XrkqwtJGRlroz5hIeAp0Zvpw2qVo6/omK8vIMTuGbblG0Gpqwd1Iv5pQVJs67pAEX5WNP6+PnUen+nTNlfmb9grXADmqpJHfYMq73SolFKxH9WLdMsCNJXWz84liGPZaT3/vUS3j4gDuRJUjGSFbxjBo8r+VjqiFCb39Nlv9In3qZlQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, 21 Feb 2025 14:16:31 +0000 Thomas Prescher wrote: > On Fri, 2025-02-21 at 13:52 +0000, Matthew Wilcox wrote: > > I don't think we should add a command line option (ie blame the > > sysadmin > > for getting it wrong).=A0 Instead, we should figure out the right > > number. > > Is it half the number of threads per socket?=A0 A quarter?=A0 90%?=A0 I= t's > > bootup, the threads aren't really doing anything else.=A0 But we > > should figure it out, not the sysadmin. >=20 > I don't think we will find a number that delivers the best performance > on every system out there. With the two systems we tested, we already > see some differences. >=20 > The Skylake servers have 36 threads per socket and deliver the best > performance when we use 8 threads which is 22%. Using more threads > decreases the performance. >=20 > On Cascade Lake with 48 threads per socket, we see the best performance > when using 32 threads which is 66%. Using more threads also decreases > the performance here (not included in the table obove). The performance > benefits of using more than 8 threads are very marginal though. >=20 > I'm completely open to change the default so something that makes more > sense. From the experiments we did so far, 25% of the threads per node > deliver a reasonable good performance. We could still keep the > parameter for sysadmins that want to micro-optimize the bootup time > though. I'm all for auto-tuning but yeah, for a boot-time thing like this we require a boot-time knob. As is often (always) the case, the sad thing is that about five people in the world know that this exists. How can we tell our users that this new thing is available and possibly useful to them? We have no channel. Perhaps in your [2/2] we could be noisier? =20 HugeTLB: allocation took 4242ms with hugepage_alloc_threads=3D42 and with a facility level higher than KERN_DEBUG (can/should we use pr_foo() here, btw?). That should get people curious and poking around in the documentation and experimenting.