From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B748FC4332F for ; Fri, 3 Nov 2023 09:56:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 307DA280017; Fri, 3 Nov 2023 05:56:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B7BA28000F; Fri, 3 Nov 2023 05:56:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17F43280017; Fri, 3 Nov 2023 05:56:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0935A28000F for ; Fri, 3 Nov 2023 05:56:07 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id CEC581CBBC2 for ; Fri, 3 Nov 2023 09:56:06 +0000 (UTC) X-FDA: 81416187132.07.0DE5099 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf26.hostedemail.com (Postfix) with ESMTP id D033714000E for ; Fri, 3 Nov 2023 09:56:03 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=Boti3rzX; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf26.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699005364; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ePm7WjwbNkGtg7aZPPVJIXKZCUCupTKzceDuad9x8Qg=; b=pnwcGE5Ai+8nLiHBoNZeCEC/ZgpwdUMvlbLAGrXuJ7WCqzcZLQCO3qQwlJkTk6mup3JBKL FA2xZJ7Gp+Dbp/2iiHt3DRKVBv+M2UOA5hQbbqlZPHQFwgrP7TQv3589Mnh+xGSa5dLWk4 t5GVLLzz8gxWOAZ0IBhJ/cAEWwNnJ+0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=Boti3rzX; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf26.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699005364; a=rsa-sha256; cv=none; b=zh3bwdR/MiDVEpXpj2D8NY7MQuV2+5XzAlUn6nP//85u0iyQoe1uV76N5oovcL7ZqtjemK MViy4zwQHIiQq8yfe/T1xtByNGYf8ZW7cD04TiyWg5scWPZZiPRHe9k6F65c1pvQ7qdPQw RZqgFCfq5jST2SXJGEOId/eMSsrHwxo= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 111421F8C0; Fri, 3 Nov 2023 09:56:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1699005362; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ePm7WjwbNkGtg7aZPPVJIXKZCUCupTKzceDuad9x8Qg=; b=Boti3rzXvIy6GpXicVmWEQvGmOFfKkcLOYxK/Vs1qx/LvOFOCNZik2CcJI4ZDtqjYPnUt9 TUx62ElcRoAqB72miNFGbGnLgblzwKs4IHCvbfpL90O8ZapU9e3zW9je11NEo5PfW0kicP FhjCCXDkC1IoRZxEJR5d3903OYs8/uU= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id DB32513907; Fri, 3 Nov 2023 09:56:01 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id AY2ENLHDRGUWBgAAMHmgww (envelope-from ); Fri, 03 Nov 2023 09:56:01 +0000 Date: Fri, 3 Nov 2023 10:56:01 +0100 From: Michal Hocko To: Gregory Price Cc: Johannes Weiner , Gregory Price , linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org, linux-mm@kvack.org, ying.huang@intel.com, akpm@linux-foundation.org, aneesh.kumar@linux.ibm.com, weixugc@google.com, apopple@nvidia.com, tim.c.chen@intel.com, dave.hansen@intel.com, shy828301@gmail.com, gregkh@linuxfoundation.org, rafael@kernel.org Subject: Re: [RFC PATCH v3 0/4] Node Weights and Weighted Interleave Message-ID: References: <20231031003810.4532-1-gregory.price@memverge.com> <20231031152142.GA3029315@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: D033714000E X-Stat-Signature: 14t3xni7zzf7yh93xgxuhkhrhab3orct X-HE-Tag: 1699005363-983380 X-HE-Meta: U2FsdGVkX1+jSh0jpr9qNQ+wpJjQwZLc2I7uAmko6IAD5fDX50VBppMR0y5grudh3wsNRINBNgROOIZHypxNVqhm/lfCCA5Bmy71tAvPs9EzuWP/xaOX4ylVNQ4dkl2Y/+AMwEySuFB+AkLJ5PbdnbPc8e1ZMwk3JbQXrkNj6+lY06CH729yOIwNbp1hDPJcQm6DWyKcI+Ri5SpvKwbV8ab9LSrFhPVm3TaekIcykVkcXW+gvT3adneQ4z5xlPwSPMAgZocgmyu/wKvRntAulkjfWqqLQXwbL51dE0kFG0aNCeOnUW5xiP5K5ZEtQqXM7aihWYI0YJnLVJUQ8d08O5iivmGedxW7S3KlQKu7O1mHZBfuWL/ELT+knoZnOZ3wL8Xs7ovv9bQmtKKTudaII7exKoYHvzrS+0+k4N2YFY21FkObYasM8YKJ21k+Fi0JxmK54+NH9tkiBPYNYWVaOkJkQRMM1rmvX7fiVenIi588TN/pFahlb7WjInLxD0selQ8q34DySWGLB1XlzMJHwsRHQkKLM3Ou4yAsJIoVIcztFH290L+lM7fLsKwWqSjxMeUzLu6YkjI6qYK0Iv4NLcafuAVSc5qKz7XPp2HF2/rEi/gxrpPewDvtz7KrdQmqRWNs9lRfMqrmEHSlqjsLkCUUJLkenG2YSp3fYjboGkKeHSVfZu3vd9gB/b/uWJU0eX5g1StU8ogRxoIApVGH3AkndbWtxG6KN3vj4XPqCSSe2jU/TjcdRrj/A5tCNf/8GFIV5WFFIAxB90V74bBTmzzHjoN6KPnKNw5KuzdVUz8a4qa42PuBYd0fOq0HcwE7zVbpB/Y0SpEQYDwTJkmCMBsqSSMGOIW5Eaq36GVSR2Vbocca2UTuf3AgIUN5IFk98jMOEjpFVj4x5Pxq9ed8HSDaAHwOwP4oHCEjwqq0R7rMLMRM/prRsy/Fwt7k+vPx/k26uox7PY7Oubuxaxc 1yh5yDt8 TaBUrJaFqmmDK5ofpnwDFbKE2SqcNvmoo2C3RZPsKZVLHBz0QYyXIf8hL9nI16EF0u8u8L+XmL6H3ww0BVOJ59OBk+ueVLQr+1HJEemADxZMBOqZffQAHQxtpwryEv3+0dP+xXA33mmCs5uBnRqeMRWso+ROfUqTX90pQ3vchEJTPg1mUOn+BPd/6vIhcQH1fxRobE5dnBzAKTmJKP+f89wfPVrqCikoXRwl4GsXtBcpQxsXOHM8KOPHijvY5OwsxT+vo14mcx6FJP6ikU/ci2A4dewIHkXfYksKPhz0XbXPo2W/9Hf1VK5Ni4oDIRs0kjfLY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed 01-11-23 23:18:59, Gregory Price wrote: > On Thu, Nov 02, 2023 at 10:47:33AM +0100, Michal Hocko wrote: > > On Wed 01-11-23 12:58:55, Gregory Price wrote: > > > Basically consider: `numactl --interleave=all ...` > > > > > > If `--weights=...`: when a node hotplug event occurs, there is no > > > recourse for adding a weight for the new node (it will default to 1). > > > > Correct and this is what I was asking about in an earlier email. How > > much do we really need to consider this setup. Is this something nice to > > have or does the nature of the technology requires to be fully dynamic > > and expect new nodes coming up at any moment? > > > > Dynamic Capacity is expected to cause a numa node to change size (in > number of memory blocks) rather than cause numa nodes to come and go, so > maybe handling the full node hotplug is a bit of an overreach. > > Good call, I'll stop considering this problem for now. > > > > If the node is removed from the system, I believe (need to validate > > > this, but IIRC) the node will be removed from any registered cpusets. > > > As a result, that falls down to mempolicy, and the node is removed. > > > > I do not think we do anything like that. Userspace might decide to > > change the numa mask when a node is offlined but I do not think we do > > anything like that automagically. > > > > mpol_rebind_policy called by update_tasks_nodemask > https://elixir.bootlin.com/linux/latest/source/mm/mempolicy.c#L319 > https://elixir.bootlin.com/linux/latest/source/kernel/cgroup/cpuset.c#L2016 > > falls down from cpuset_hotplug_workfn: > https://elixir.bootlin.com/linux/latest/source/kernel/cgroup/cpuset.c#L3771 Ohh, have missed that. Thanks for the reference. Quite honestly I am not sure this code is really a) necessary and b) ever exercised. For the former I would argue that offline node could be treated as completely depleted one. From the correctness POV it shouldn't make any difference and I am rather skeptical it would have performance improvements. And for the latter, full node offlines are really rare from experience. I would be interested about actual real life usecases which do that regularly. I do remember a certain HW vendor working on a hotplugable system (both CPUs and memory) to reduce downtimes cause by misbehaving CPUs/memoryu. This has turned out very impractical because of movable memory requirements and also some HW limitations (like most HW attached to Node0 which has turned out to be single point of failure anyway). [...] [...] > > Moving the global policy to cgroups would make the main cocern of > > different workloads looking for different policy less problamatic. > > I didn't have much time to think that through but the main question is > > how to sanely define hierarchical properties of those weights? This is > > more of a resource distribution than enforcement so maybe a simple > > inherit or overwrite (if you have a more specific needs) semantic makes > > sense and it is sufficient. > > > > As a user I would assume it would operate much the same way as other > nested cgroups, which is inherit by default (with subsets) or an > explicit overwrite that can't exceed the higher level settings. This would make it rather impractical because a default (everything set to 1) would be cast in stone. As mentioned above this this not an enforcement limit. So I _think_ that a simple hierarchical rule like cgroup_interleaving_mask(cgroup) interleaving_mask = (cgroup->interleaving_mask) ? : cgroup_interleaving_mask(parent_cgroup(cgroup)) So child cgroups could overwrite parent as they wish. If there is any enforcement (like a cpuset) that would filter useable nodes and the allocation policy would simply apply weights on those. -- Michal Hocko SUSE Labs