From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3CDEC4707B for ; Thu, 11 Jan 2024 09:25:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2198F6B008C; Thu, 11 Jan 2024 04:25:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C8D56B0093; Thu, 11 Jan 2024 04:25:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 092F46B0095; Thu, 11 Jan 2024 04:25:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id EA0F16B008C for ; Thu, 11 Jan 2024 04:25:12 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BFBC91A04BC for ; Thu, 11 Jan 2024 09:25:12 +0000 (UTC) X-FDA: 81666496464.29.DA8350F Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) by imf14.hostedemail.com (Postfix) with ESMTP id D7B1B100009 for ; Thu, 11 Jan 2024 09:25:10 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=fromorbit-com.20230601.gappssmtp.com header.s=20230601 header.b=KwGJSLxt; dmarc=pass (policy=quarantine) header.from=fromorbit.com; spf=pass (imf14.hostedemail.com: domain of david@fromorbit.com designates 209.85.167.174 as permitted sender) smtp.mailfrom=david@fromorbit.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704965111; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oPRsV3mpGhgl4tIUNnxd2xjVM8/07Y43tjO1bvwG0Q8=; b=FH//HxbXLNX2n7RIvFzwtplw9dOHKzMLXTvXy+lge9ZI4uUidyNFoZAaZCA5JDwEg3eV2l PYNjV7QmTLkZ6qKllbfokXxWVx/M0bCq6wlIW/sFRY8JyVa4vMgkd9MvrX9bX7DQon1dKZ WeTmhFvJ4mV0D6zL0rMb808sdSTC2Ic= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=fromorbit-com.20230601.gappssmtp.com header.s=20230601 header.b=KwGJSLxt; dmarc=pass (policy=quarantine) header.from=fromorbit.com; spf=pass (imf14.hostedemail.com: domain of david@fromorbit.com designates 209.85.167.174 as permitted sender) smtp.mailfrom=david@fromorbit.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704965111; a=rsa-sha256; cv=none; b=mKvKapVveCMfnANUYeuqfv//fk8oQw/aKqa76s/x0uQ+i6azbKiEdBnzwc0fPn60vFP9lV cu7N6KD/X2f5sqsRU1PlsyOFPizzAVNrlSnKUm1n97tjrhar5Lz/yg9g2CsWom/ctgsPrF pre+QQUExmkx03ViR3UXPkioqZuV4lA= Received: by mail-oi1-f174.google.com with SMTP id 5614622812f47-3bbc5636b8eso3206014b6e.2 for ; Thu, 11 Jan 2024 01:25:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20230601.gappssmtp.com; s=20230601; t=1704965110; x=1705569910; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=oPRsV3mpGhgl4tIUNnxd2xjVM8/07Y43tjO1bvwG0Q8=; b=KwGJSLxtojQ7C5/dsccl4ANF4gMj9Gh3i2VioQyNJ1SumM84iNQVC2L80TKSxQxXzC uAmuhcXldn4s1UXdjaNNAW16KYr3nhUwgMZVJcqitrMl+afsUQcNSL22YFWuT1d+0K8h q+k+BfjUMkwIeCC16xHqHqwH4gHmM5zOvW/eNNS0hS7OqaxhJsLT7kmk4dvPyNHVoWkh jm5GzNLY465Znesp1JL3Vk2lB1PmzA9436YBQajhKsbYxbcjb+7qE8W5QLyUmQolG2fO dJapJHB5HDIwV/YF3TKu068eRKoYVer8gdvwo8pQ1gKjdUBt9ZiTmhJ5JVfxRbq851mU +/OQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704965110; x=1705569910; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=oPRsV3mpGhgl4tIUNnxd2xjVM8/07Y43tjO1bvwG0Q8=; b=K8hFAhDOrbdFzYnf5IOzKNkE9GrRLNyXUMXVlWH4ZwtT+Jtg90lI6mAdj0mwj+80Wt yM5VbMAN+oI1V4wsYIHYHY6h5TSkxZ81Pp38dxOkegCNXcHP5Nl1JtDVOXrzTsTLgoiX J3QCZC+1+N81pF7/oek/rBD4QWSAIOqs9eJVOzDE1oqn+RYvVf68g3LzEBcAwHogGvAA ybBtu3HPCPx3uOYc0uvwM/IcfjELe48aQA7m+22dhwJgQCyXJCDiHOJYnF1Mp5I1qcYH g4qJ03Pa7VWP5iB/0fM68QFdpTNmZQ6S+i2ZyiRbsyRlzK/wYbdhKHL6AyBT3RUHkgeF eSUw== X-Gm-Message-State: AOJu0YyGmfRQC7sGpMuJXr5TOXJbmLR9GXiUAER5MLfoGCUU0myzdJwm 4zD2aRSs9CMxF1fb1fGWJdSHYkI4lvP477VIA0AtfwWIBcI= X-Google-Smtp-Source: AGHT+IEZJkHdKjVYcCRHPRa0m7EgOIyHW6Fngfug8G8Oi8WGZmvzxa4qkvyzCbfiIZPptwktvdeUhw== X-Received: by 2002:a05:6808:1782:b0:3bd:3db7:402d with SMTP id bg2-20020a056808178200b003bd3db7402dmr1032846oib.89.1704965109927; Thu, 11 Jan 2024 01:25:09 -0800 (PST) Received: from dread.disaster.area (pa49-180-249-6.pa.nsw.optusnet.com.au. [49.180.249.6]) by smtp.gmail.com with ESMTPSA id fa2-20020a056a002d0200b006d9af8c25easm708198pfb.84.2024.01.11.01.25.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jan 2024 01:25:09 -0800 (PST) Received: from dave by dread.disaster.area with local (Exim 4.96) (envelope-from ) id 1rNrJD-008tBh-02; Thu, 11 Jan 2024 20:25:07 +1100 Date: Thu, 11 Jan 2024 20:25:06 +1100 From: Dave Chinner To: "Uladzislau Rezki (Sony)" Cc: linux-mm@kvack.org, Andrew Morton , LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , "Paul E . McKenney" , Joel Fernandes , Oleksiy Avramchenko Subject: Re: [PATCH v3 10/11] mm: vmalloc: Set nr_nodes based on CPUs in a system Message-ID: References: <20240102184633.748113-1-urezki@gmail.com> <20240102184633.748113-11-urezki@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240102184633.748113-11-urezki@gmail.com> X-Rspamd-Queue-Id: D7B1B100009 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: h3s36by6gewy336fkqcd59rz9x83iyns X-HE-Tag: 1704965110-236365 X-HE-Meta: U2FsdGVkX18YUmmsyzWtWxcJfxDvBIuJt04HLXWDoz3cy1LOuUND849UFh7NsN7pG7injKNzio8fXpJKmLgd3Sy7fEnzSm6H20JjqOqvAn6jtLAGmgzZciFE+QBSrLsXNCfSUmMlublnWF7lgPfcsyiSawrMuaPc5bhcIiqjadtf6p3gGiKhcW0djmmt0+6HuPOZAjWKKfuHs1y8EDkYvNK6yhtjLRrvgVAinVrr9AEhyMB622v32r+l0TlNUAyN3PKC4CrSbggZJ6mtGnGzJxMY47ob1u/Pcqm83UDgw7s68bz9H5nqRg2Lo9RcwZNbfQx4KBT4z2hhuUGhEZiWzJJDxAiMKa76o1MvnCS+Hw0l0nOGVuEJW7Ie8qATZnEuCmWMIicKr1PUFOgYXPG3SjJoDcNWNsk9Us+85szeE8NYZMTDrhsy2uW7+ruu01LA6gJx/ihbwjugiJXuLbaT1j4qlaZxTIJvq5ohDItFFzxH2JVVYVjeRtaCXJY+3hDGaVMKOezcj3/FOEYOhfmC95MkCjT2u6LDUt9ADAU4M60vh9gw9bC9CW9vAURQQqcmSk5Ub6uIV5cwhWiCMy5u6yMg4QiGZ0S71brwEjq8OG1lmbH60D5V+wV0sqXtfFnSaGK+aRGl0kfurXoUWvvKB2l/3Mxon/Oq+CDJD6sRcKvCFyyC+6IEhFgivmqZLmP7kvY1h/CfVpw75fw8brAYqaYVyhra2ZbBKQTLW1TTGlg0tVygHJMFiYh1LtTYd4WHy6lOCBtfU4Jqyeg+uI0rBcoKXpmKG23dh5tcPvXtVVJAPbgqINnWfrJFIzOog/eoFX3xTwb0JD+KSoxQrr7eU7czBYX5utdvvXthMAjpe4nukHtLFKh/CSgH1v7YdHxhFimY+bdB3rM/ZrYaCa2OyocUa8wGYRVq5TuxoVDOM13cxz6Lx52EkFRqaG/0NNCdf8ItYh/3lABW0LdKkGq dbfrs7pI I7auFdJrB2V6ct+6UO6UZoCFUiAn8GuzxEt5yXZfv9GLNqQArZkbfoh5CSoq2QnWxS0GNheJbN8ANyhs/b6f7nU8juduWcm5aKAEzKctybAkRbeMrYzZPS26Q9ZCiuz4YM4HkM7A6+Z89U0OBGt8WUNiVqgJ3cRSYhNcdqGSLd2BkUcu3MpmQEwOfMVSkVuKvlpafxh0GTx26g9OCoqq8PmImG/gsO4h27b9dx3g9HMONYZScnbs8O1ryLlq0Mk/1fhoGBGo+ZFNP/SAyiaKffnNW81tYBdHpuRWkTgPSs4oR3UNtgaHzTb0xjcAiwEMLj5R+yl9J83jT3+SKeyM6r2+B7RRrAyujaYt3Lwd9OwvkSl9ByEcrRfs3MYeUZ4PRQbMYxn4abzMm2Xdi3ml0xsq/LTcWWYgqSyevgU1ExvTxavYikMVKTq/c6M9MOT2gznneVAZyDH0cO/S/Cudl6bh+IGXeI5jFbb9Cbd5ARaeMYN+y43OTax9tBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jan 02, 2024 at 07:46:32PM +0100, Uladzislau Rezki (Sony) wrote: > A number of nodes which are used in the alloc/free paths is > set based on num_possible_cpus() in a system. Please note a > high limit threshold though is fixed and corresponds to 128 > nodes. Large CPU count machines are NUMA machines. ALl of the allocation and reclaim is NUMA node based i.e. a pgdat per NUMA node. Shrinkers are also able to be run in a NUMA aware mode so that per-node structures can be reclaimed similar to how per-node LRU lists are scanned for reclaim. Hence I'm left to wonder if it would be better to have a vmalloc area per pgdat (or sub-node cluster) rather than just base the number on CPU count and then have an arbitrary maximum number when we get to 128 CPU cores. We can have 128 CPU cores in a single socket these days, so not being able to scale the vmalloc areas beyond a single socket seems like a bit of a limitation. Scaling out the vmalloc areas in a NUMA aware fashion allows the shrinker to be run in numa aware mode, which gets rid of the need for the global shrinker to loop over every single vmap area in every shrinker invocation. Only the vm areas on the node that has a memory shortage need to be scanned and reclaimed, it doesn't need reclaim everything globally when a single node runs out of memory. Yes, this may not give quite as good microbenchmark scalability results, but being able to locate each vm area in node local memory and have operation on them largely isolated to node-local tasks and vmalloc area reclaim will work much better on large multi-socket NUMA machines. Cheers, Dave. -- Dave Chinner david@fromorbit.com