From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF84DC4828F for ; Wed, 7 Feb 2024 10:18:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7DA436B0074; Wed, 7 Feb 2024 05:18:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 73BEE6B0075; Wed, 7 Feb 2024 05:18:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58E7C6B0078; Wed, 7 Feb 2024 05:18:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 40D696B0074 for ; Wed, 7 Feb 2024 05:18:32 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D9A38C0362 for ; Wed, 7 Feb 2024 10:18:31 +0000 (UTC) X-FDA: 81764608422.20.CCF5B21 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id DC85C180002 for ; Wed, 7 Feb 2024 10:18:29 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707301110; a=rsa-sha256; cv=none; b=BkM8QqCKEzFVUhGeo/gsSjI2ctxC941HTYespOpZ3POXy0FHC/aKjsWoEv2yvIaYX618lJ fU91Yt9f1Ty64T7mS9rESdPg0JqYDgzeI1iMUv/BErw20S2b6DQIcPVbD8bhjCo06ddC1d 72ilAoGXGF2yTo0NggicDHNtGc9FtFg= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707301110; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tl6bNa+d0puBkbJhNwcO3JKxL43EYRPkfL04Em8SKls=; b=44DjgukA1eYqq89Pbutwz9v52fsNg6yWYls2H9qi2W6KQS+mRyYFEe4DyDAhkCgKoNNBRp ur7MjFUAb8QzYn7R6cjBP9oa5pieK7Ev/lo7zMhquFSyCsX73OU5e0QGzh0nY53nElbfkm VnU+TuVTCAL89LoTfohbeGXUd/TUCvg= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 28B86DA7; Wed, 7 Feb 2024 02:19:11 -0800 (PST) Received: from FVFF77S0Q05N.cambridge.arm.com (FVFF77S0Q05N.cambridge.arm.com [10.1.26.150]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 94E4B3F7F4; Wed, 7 Feb 2024 02:18:26 -0800 (PST) Date: Wed, 7 Feb 2024 10:18:24 +0000 From: Mark Rutland To: "Christoph Lameter (Ampere)" Cc: catalin.marinas@arm.com, Will Deacon , Jonathan.Cameron@huawei.com, Matteo.Carlini@arm.com, Valentin.Schneider@arm.com, akpm@linux-foundation.org, anshuman.khandual@arm.com, Eric Mackay , dave.kleikamp@oracle.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux@armlinux.org.uk, robin.murphy@arm.com, vanshikonda@os.amperecomputing.com, yang@os.amperecomputing.com Subject: Re: [PATCH REPOST v2] ARM64: Dynamically allocate cpumasks and increase supported CPUs to 512 Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DC85C180002 X-Stat-Signature: hjn7wepeqf39cp4gzkobkex76t3ya6dw X-Rspam-User: X-HE-Tag: 1707301109-182284 X-HE-Meta: U2FsdGVkX1++ca9c/B4A5ti+uW/g3zero9J/LJM4xg3/e6E/lHuWZ3NA+ivIWN69q12mURvOKguSfPlJybOSSKc1+y8SqDtSalwpPxOl+LDfXDNHXJxA3/RT8t1yIfYFAcqHUW6Qxf+i2nJWYYTzFk2APbmpG27Ahx82YX6jEhlIgm8ruPdJfJUuE6uV/0cC1P5RY3n9rNBDs7vf298lcQN5nsXsDzB6DnBo9So+23XiqsgPBZaoilhVR0rmC88/uIG2RDQ8RRwdK8SyFebEhk4xwf2zsBXS/l78AUMuT3Y2Btv5hyXqsdb1YuFvJrmaT4ojo/3eLOluT4Bxb02De2N2SVVFfozxIQtOeS3JguzSfp5Ek0RAkJ2Yg1IkwvOFTaiaMS3kiC08FuZEL1AK8A2tb4sWcxxEqgkRq1gBs7yUr/SmtuVmepxm4esDeFi+5QUKESDaU3qxPderefvPSw9ScCf7qqug87JeR+s+DRJswR5yxD+UG2+6JVvtaFfhowaTgkhf7A+ZSAFhTJhS0QisFS2Qo1ZRFKdD+hsQg/Vg0ajLvPRCwPoiXnWAmf9NvUrSkt1DryyULUjKnOZXu1PN5hhgSTRWbKeAyCh3a4A9WnxKg+w2NwaNepSuBCJwojY88PxLQnb0lSdLUDWViTUUJa/SjS1QGD45QhxGG/pxZZWrrb30bBQ4lUC0CRP6fEIhVVqOkQ3z3GXKbqnYrhbZIh8sqsAHKcyhLRpDscqLUJQfvjAf0U73LESUjOapwKXINaJeWQ1YyxDzAJNWItqM8OrGGqU822uRssNOBowrlm4C04CMHY126fS6HnARAjWXw+SgnTmRCVeFtHOD8bl7NmvQbmGFyYvz1rHSoOb10vdTO8hbevOu4bHNG3AZCGQ1Wqa8rYM+W24vO1awbT8CXWr0lNuuBxVc+u8ZfaD5Br5inrRfbSHYr5GE+4ZuchpaMAok9cxAgY3rrwD XgiP6yEe GtLVwgaQEfvq3O0JsG/yNtZnf/CTqRarK6SJ+fC+zBcL4deDtyGUOOH2sUj+1L/ll4PlysVpc8ALigTp9CcWQrLmI9JtcewceJLepq2ox1IQeBye/Slk14Ybm2G27ewtYmKi/8HmaMXUvQ8eRlI8wDoHf1gmnIbpfjrNPF3xxnSCCca1N3hWre9OlAbuIvnx5ETZ3v+pXY8RuRtrF88ZzfKW5aA7Hf7211ZD9zVi/J/7BvGeyyzwRxf2H9XJnx8+ntemo7YLC61aSsDbLheaWRDcYL1x44VMIWrnGiIJ/kVogfjyfV6FW1lXVqI4cT4IF1w6J6XF5GFL+T3UDgwElszGg2DntNFtbg8KoBI/8Yq0oHBA8DG+WzfRspVm8HzcXQH+3Cw5avlFqXwis352khcU7YNvxvGK3cgOo/VxueLajpZzjtsZqi82YHgXeYOXBzNscnjdlSSXhd7Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi CHristoph, On Tue, Feb 06, 2024 at 10:09:59AM -0800, Christoph Lameter (Ampere) wrote: > Can we get this merged for 6.9? The patch has been around for awhile now. > > Ampere Computing develops high end ARM processor that support an ever > increasing number of processors. The default 256 processors are > not enough for our newer products. The default is used by > distros and therefore our customers cannot use distro kernels because > the number of processors is not supported. > > One of the objections against earlier patches to increase the limit > was that the memory use becomes too high. There is a feature called > CPUMASK_OFFSTACK that configures the cpumasks in the kernel to be > dynamically allocated. This was used in the X86 architecture in the > past to enable support for larger CPU configurations up to 8k cpus. > > With that is becomes possible to dynamically size the allocation of > the cpu bitmaps depending on the quantity of processors detected on > bootup. > > This patch enables that logic if more than 256 processors > are configured and increases the default to 512 processors. > > Further increases may be needed if ARM processor vendors start > supporting more processors. Given the current inflationary trends > in core counts from multiple processor manufacturers this may occur. Can we please make this simpler, and clearer e.g. Currently defconfig selects NR_CPUS=256, but some vendors (e.g. Ampere Computing) are planning to ship systems with 512 CPUs. So that all CPUs on these systems can be used with defconfig, we'd like to bump NR_CPUS to 512. Therefore this patch increases the default NR_CPUS from 256 to 512. As increasing NR_CPUS will increase the size of cpumasks, there's a fear that this might have a significant impact on stack usage due to code which places cpumasks on the stack. To mitigate that concern, we can select CPUMASK_OFFSTACK. As that doesn't seem to be a problem today with NR_CPUS=256, we only select this when NR_CPUS > 256. There's no need for all the other gunk. However, can you please comment on the impact of this? e.g. * Does it make the kernel Image or vmlinux any bigger? If you can build the kernel before/after this patch and dump the output of 'ls -l arch/arm64/boot/Image' and 'size vmlinux', that would be great. * Does this have any obvious impact on the memory usage of the kernel? * Does this have any obvious performance impact? e.g. is a kernel build any slower/faster before/after this patch on a system with 256-or-fewer CPUs? > Tested-by: Eric Mackay > Signed-off-by: Christoph Lameter (Ampere) > > --- > > Original post: https://www.spinics.net/lists/linux-mm/msg369701.html > > V1->V2 > > - Keep quotation marks > - Remove whiltespace damage > - Add tested by > > > Index: linux/arch/arm64/Kconfig > =================================================================== Can you please use git format-patch? This is more painful than necessary to apply with usual tools like b4 and git am, and the arm64 maintainers are much more likely to pick up a patch when they don't have to go outside of their usual workflow to do so... > --- linux.orig/arch/arm64/Kconfig > +++ linux/arch/arm64/Kconfig > @@ -1407,7 +1407,21 @@ config SCHED_SMT > config NR_CPUS > int "Maximum number of CPUs (2-4096)" > range 2 4096 > - default "256" > + default "512" > + > +# > +# Determines the placement of cpumasks. > +# > +# With CPUMASK_OFFSTACK the cpumasks are dynamically allocated. > +# Useful for machines with lots of core because it avoids increasing > +# the size of many of the data structures in the kernel. > +# > +# If this is off then the cpumasks have a static sizes and are > +# embedded within data structures. > +# This described the semantic of CPUMASK_OFFSTACK, which is not specific to arm64. If we need a comment, it should explain *why* we select this for more than 256 CPUs specifically. I think we can just delete this comment and rely on having that rationale in the commit message. We can find that with git-log and git-blame. With this comment gone, the patch itself looks fine to me, but as above the commit message needs to be cleaned up and it'd be *really* helpful if you could send this a git formatted patch. Mark. > +config CPUMASK_OFFSTACK > + def_bool y > + depends on NR_CPUS > 256 > > config HOTPLUG_CPU > bool "Support for hot-pluggable CPUs" > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel >