From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2F7DC54E4A for ; Thu, 7 Mar 2024 17:49:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DC406B024C; Thu, 7 Mar 2024 12:49:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48C066B024D; Thu, 7 Mar 2024 12:49:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 355746B024E; Thu, 7 Mar 2024 12:49:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 21C926B024C for ; Thu, 7 Mar 2024 12:49:23 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E424A1611F5 for ; Thu, 7 Mar 2024 17:49:22 +0000 (UTC) X-FDA: 81870979764.13.5A0A045 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id E8F4E10000B for ; Thu, 7 Mar 2024 17:49:19 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709833760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ue1Up5mSVwKf7p5lM2ArNsNd0UF1JYZ1ROzp3TojLBg=; b=jSeBxb56u5Iq69B8M19JaPFGMR77rP/6sTFPFRapBou9HGSc2VshoX9lWiZxvxwBn8//Xd Xh4L8VHyzHOXBB8Rdcz4qRbftSSq3D+nt+haLZrvx0vVxxZNJGIoJxAfFcr1ZcTsbZGgND nuOEQOWolHPbWRqoI7qdkjQQKtXIdmI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709833760; a=rsa-sha256; cv=none; b=6X01MAfEDCTCDPWcKU32NMadp8KbekJErtR0Xjd3H6lD+bEMn5LY25Atl4Swu3CxgfIn1z 6t4/CRwCTu28g4UesBhwOIQrfNwAJtDj45GFcqnEysyxqjN4Px47hh2mV9WaKAG3hTJKMr OwzksWI+Yvs378Oo8ddkeqH8RRawTy8= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BCCF11FB; Thu, 7 Mar 2024 09:49:55 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.69.155]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D7D073F762; Thu, 7 Mar 2024 09:49:15 -0800 (PST) Date: Thu, 7 Mar 2024 17:49:12 +0000 From: Mark Rutland To: "Christoph Lameter (Ampere)" , catalin.marinas@arm.com Cc: Will Deacon , Jonathan.Cameron@huawei.com, Matteo.Carlini@arm.com, Valentin.Schneider@arm.com, akpm@linux-foundation.org, anshuman.khandual@arm.com, Eric Mackay , dave.kleikamp@oracle.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux@armlinux.org.uk, robin.murphy@arm.com, vanshikonda@os.amperecomputing.com, yang@os.amperecomputing.com Subject: Re: [PATCH v3] ARM64: Dynamically allocate cpumasks and increase supported CPUs to 512 Message-ID: References: <37099a57-b655-3b3a-56d0-5f7fbd49d7db@gentwo.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <37099a57-b655-3b3a-56d0-5f7fbd49d7db@gentwo.org> X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: E8F4E10000B X-Stat-Signature: dsuukgau79fhrjzztnkoqoi4i3sx9b8g X-HE-Tag: 1709833759-606676 X-HE-Meta: U2FsdGVkX1/W9lg47A08flzLQTQH1buIV9Zw6oxEE+0r3RRgz6RFWF52mSKo8pE5AxQmbX7rR/tEvmQAylqzu2Tzk9MuuKegdfZmloMOqoVHtWuNom9s6Xp+D+/bNomdtf7jG6XHVkt30QeuetCgvUL6TqhjVw2ncVKfIEUU75BGDoPDxxQktBqIScK7YXsQY/RM0Vab0tPXQxo0o6IL0mNpYg+CS/jEdAE+JWCnqO5Z9u9I0bCcADBty6g9qgkjG5/mWM5MyU5B5pvMRZ0ZSeqpxUClAWHTsFX5+8zidG7oA31DgAtRHyfF01Yp/wJWlyRV4s3D54RbR8xiDT0ghbY9HenwXgN9t/uJRY4kkQbZNceOh0otxmmzZIbrJRGXCrZvmUNPRqhshBEhw3l7/gAG9ZzALzSmVd24U2NzEcDrd2fPQe+mrruQir4Wufk811UvOJ6GSPOzwMejTAC2Wgqih9Pgwhp5n49vzUFTTM1ttq2k0n2NPS19HiYrPwXBfLTfMvZi31+cfSdPJW8kxiwXioN6s8UOapTMuIHIcPNuV7MhycKV/di3Twp+0BLbN9uLCBq8hUCQ7KzOETCrn1y4efhhRcBNWfKbF5gVTsnOQmhJlThb6lEtUNHUG/RNCGpzf9L5u+vimx9p6Olg/mfmI5l/QcPtOrydv84v0Jncmo0lmeb68RfEGFUiPC9hUnwj7jKDUfOLgd05rwExQRGlQs/2PX/xSIwrF24d+0WKyGGQL/EVgyHtovK5Gpg3sC4NF2zYKkJGKwBRJTMC0spA/NCI0xxz8RDYMPw5Uk4LNXw8Kwb6WQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Christoph, On Wed, Mar 06, 2024 at 05:45:04PM -0800, Christoph Lameter (Ampere) wrote: > Currently defconfig selects NR_CPUS=256, but some vendors (e.g. Ampere > Computing) are planning to ship systems with 512 CPUs. So that all CPUs on > these systems can be used with defconfig, we'd like to bump NR_CPUS to 512. > Therefore this patch increases the default NR_CPUS from 256 to 512. > > As increasing NR_CPUS will increase the size of cpumasks, there's a fear that > this might have a significant impact on stack usage due to code which places > cpumasks on the stack. To mitigate that concern, we can select > CPUMASK_OFFSTACK. As that doesn't seem to be a problem today with > NR_CPUS=256, we only select this when NR_CPUS > 256. > > CPUMASK_OFFSTACK configures the cpumasks in the kernel to be > dynamically allocated. This was used in the X86 architecture in the > past to enable support for larger CPU configurations up to 8k cpus. > > With that is becomes possible to dynamically size the allocation of > the cpu bitmaps depending on the quantity of processors detected on > bootup. Memory used for cpumasks will increase if the kernel is > run on a machine with more cores. > > Further increases may be needed if ARM processor vendors start > supporting more processors. Given the current inflationary trends > in core counts from multiple processor manufacturers this may occur. > > There are minor regressions for hackbench. The kernel data size > for 512 cpus is smaller with offstack than with onstack. > > Benchmark results using hackbench average over 10 runs of > > hackbench -s 512 -l 2000 -g 15 -f 25 -P > > on Altra 80 Core > > Support for 256 CPUs on stack. Baseline > > 7.8564 sec > > Support for 512 CUs on stack. > > 7.8713 sec + 0.18% > > 512 CPUS offstack > > 7.8916 sec + 0.44% > > Kernel size comparison: > > text data filename Difference to onstack256 baseline > 25755648 9589248 vmlinuz-6.8.0-rc4-onstack256 > 25755648 9607680 vmlinuz-6.8.0-rc4-onstack512 +0.19% > 25755648 9603584 vmlinuz-6.8.0-rc4-offstack512 +0.14% Thanks for this data; I think that's a strong justification that this isn't likely to cause a big problem for us, and so I thing it makes sense to go with this. I have two minor comments below. > Tested-by: Eric Mackay > Reviewed-by: Russell King (Oracle) > Signed-off-by: Christoph Lameter (Ampere) > --- > > > Original post: https://www.spinics.net/lists/linux-mm/msg369701.html > V2: https://lkml.org/lkml/2024/2/7/505 > > > V1->V2 > > - Keep quotation marks > - Remove whiltespace damage > - Add tested by > > V2->V3: > - Add test results > - Rework descriptions > > > arch/arm64/Kconfig | 16 +++++++++++++++- > 1 file changed, 15 insertions(+), 1 deletion(-) > > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > index aa7c1d435139..4e767dede47d 100644 > --- a/arch/arm64/Kconfig > +++ b/arch/arm64/Kconfig > @@ -1427,7 +1427,21 @@ config SCHED_SMT > config NR_CPUS > int "Maximum number of CPUs (2-4096)" > range 2 4096 > - default "256" > + default "512" > + > +# > +# Determines the placement of cpumasks. > +# > +# With CPUMASK_OFFSTACK the cpumasks are dynamically allocated. > +# Useful for machines with lots of core because it avoids increasing > +# the size of many of the data structures in the kernel. > +# > +# If this is off then the cpumasks have a static sizes and are > +# embedded within data structures. > +# > + config CPUMASK_OFFSTACK > + def_bool y > + depends on NR_CPUS > 256 As before, can we please delete the comment? That's the general semantic of CPUMASK_OFFSTACK, not why we're selecting it. That aside, this config option is defined in lib/Kconfig, so we should select it rather than redefining it. i.e. this should be: select CPUMASK_OFFSTACK if NR_CPUS > 256 Sorry for not spotting that before. With those changes: Acked-by: Mark Rutland Catalin, are you happy to fix that up when applying? Mark. > > config HOTPLUG_CPU > bool "Support for hot-pluggable CPUs" > -- > 2.39.2 >