From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77F49CF9C71 for ; Tue, 24 Sep 2024 10:43:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C7C226B0082; Tue, 24 Sep 2024 06:43:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C2BA36B0083; Tue, 24 Sep 2024 06:43:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1AFC6B0085; Tue, 24 Sep 2024 06:43:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 999ED6B0082 for ; Tue, 24 Sep 2024 06:43:36 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CA59AC1993 for ; Tue, 24 Sep 2024 10:43:35 +0000 (UTC) X-FDA: 82599295590.11.77E2111 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf14.hostedemail.com (Postfix) with ESMTP id 2DB8B100008 for ; Tue, 24 Sep 2024 10:43:34 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=L3bUtDg3; spf=pass (imf14.hostedemail.com: domain of rppt@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727174495; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=An1GhkSvkDcODdx0Gw4ctvmL8HIzrOtTwuTnFJ4gGhc=; b=iMvdQjlSuld3hIz+eY1TVhv/6OgeTOZ+iaB52NH4sZxKK9Lwt0XQIzLOYcGO3sOEqS8Hp3 91tsnmW1auk/zucXowdlXkWB8DAkCHYwxvE4wgBhQIAXx/wPDYF+6Aw6wCVF2R4TLRuqFc 0l5MyEtxVXGsMrwEbB/ew1y6rx2a9gQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727174495; a=rsa-sha256; cv=none; b=Os1psLtHaeQdiPfVlxJ5qQuVw19wgjUUF5vKCM7v3/x8MX1vFrE8AqtpvA7cpM598pyUfD RWp/4/lvu1Ztgon81KR2Zv5kBaBEJtn4Cf9119cHSl7ucNyVS5wWwQbJPyYSRmCdFc129a 4Qb7TETeAz6fb6NM5FYtYvfdGb8N2ik= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=L3bUtDg3; spf=pass (imf14.hostedemail.com: domain of rppt@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 5C17AA414A2; Tue, 24 Sep 2024 10:43:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7AA78C4CEC4; Tue, 24 Sep 2024 10:43:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727174613; bh=n6p35akJn0rRYtV1CZC9RwbESxlhP2Wfb9BUsBQBiJI=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=L3bUtDg3Ko+zNOastVUWuBFPx6lw/1Ebt07RUaJep/dat5b/31IZ+dNKvZlVOk0Kr dO/bMzrzFtaPIYa2FE1hDUuy0Pi7Zn6vr0Wa2tnyf6WG5yOVavtl/vBdpjMap9mkGU KeThdN4rY8SNEj5SK2yiZE3i6XPw8yU4u9PITACGh8yYCptTKdH4YvmFeBqMg79BrI 03pni9juOjvHZi8nDgt3BcYqw+rja6n8lwNmsmr8btZ93ChJ6ZdU75yzvd3ewQM1bm 4CFoQpz3pMIeV1KnWMCtEDNQNF8/BEWxep9/CMdZGlIPqtJAjt6kRvN1cJ0RTyfhQB EXruNGy8t9sdg== Date: Tue, 24 Sep 2024 13:40:22 +0300 From: Mike Rapoport To: Bruno Faccini Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, ziy@nvidia.com, ttabi@nvidia.com, jhubbard@nvidia.com Subject: Re: [PATCH] mm/fake-numa: per-phys node fake size Message-ID: References: <20240921081348.10016-1-bfaccini@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240921081348.10016-1-bfaccini@nvidia.com> X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2DB8B100008 X-Stat-Signature: 5d9qmo8f8wfau6gzc6byssr9xco9e1ha X-HE-Tag: 1727174614-222636 X-HE-Meta: U2FsdGVkX1+N9n0z3pwFVLbb1/AdV4CWZ/TW4cAq0QPVlr0H7yIld8HtOcHETN//XBKWzeZf8MD3qmbnG2mG2+hslbPjM0t+jwz3wm4f46DqErPy2GExB1oPYOvI5JWxiyTcVUBxVmKCyjouD6UjtinE3bcNKjM1DTi0Vht0m0Fh+10GIdTI4Lo328jvJloy70WAOLz+PpVwpCw/yA2bofw6ADZvQzUgioPifG12WmoS6o+zGVG7T6w1epiyfT12cTJYQgwE4xsvGno/p5nEgpdel4cOUXjr3LJbISFPo3icZtJCQv3osHmCSrpzB5OjvfLOjBho1e2b+5Kibe5dDXkyMB3Pc+72ks1L48cSc9uJYyjn9tnbwo5V/1B81p5dm8KLg/q2/f96Hj1i48vHvUZ2IcyJBS38N8Ubbr2GzvLn9V2hVFt4Nx0dZC9b4niOGNJxRquy3mtDnE1V87n5ZDvV0U9cwfw/xHrxfgq+OK7bn0fRUhfUcx6i6EPa4x1Egk+NAXeTCGRluOJGZPk998m7DAUrrfQ2+wyqLN10ry5RGsHsr77EFuTtEf0aJRZE2+lYxOjJmX79RVx/mPWvA/pHIlYBNl6uwiUuo+g+SlvqXbvvuw0f+ndCVa+P1a7Q4OlZcDWzb7LiuA4lFjku8rCgCMmg+eCLTl2Twib5eyWN0pvKluWb+cyeIdvLTg7+XJAbiRVM+gMVDRKwQKKWOs01xXtbJ+DGoIr1zV6kFWGg7xEsKpkV/62EvzcM8xqGGH+PKzqcG0nl3+MxJQ01F7d5ahjBxzcMQU4gbU0rkxVm1kZjfVIOhQRscxBXNs44MqVsYVbb0oYKcRYZ8dniYc7afhCJFNHs/vebCm7d7KLpMKTGlG9aI/Fhz8xHgIbb8Mar+hwtBCX/Z66vss0PqbbO56QNNwMAzg+7Hgoq2/92o11+H1qW50QxolryFN3jPun9oGaaWpU61YXIn3q Rc5YGgjO pgXkcXwqsfMGUf5bwEngEn1OLTUUot50SEzXXOkwTr6BViuj4j1amNbl/I+gTNLGiqD2bE9l985aFi7A4MP35G+z604wXBsVkSN9k943bjOqMIHbOQrSym4V5P7Pq7fifPy3LouKzJ5x126y+Q1LagCVvlNAF/ZSmC0cj5L2ZiCBJYGgG9fUcyIqxQk9UVAtvCstYKc3wW4Y+iFnXKML8oUZZzmcSvcJaZaNSQ2itHUbQ5jatlufLYqpDAbDxbSsvU16qtCEnS7JV3sqpYZeVDPXqGAR1B0WiFN5RfodjI0Ny703haSUpsOVhcg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, Sep 21, 2024 at 01:13:49AM -0700, Bruno Faccini wrote: > Determine fake numa node size on a per-phys node basis to > handle cases where there are big differences of reserved > memory size inside physical nodes, this will allow to get > the expected number of nodes evenly interleaved. > > Consider a system with 2 physical Numa nodes where almost > all reserved memory sits into a single node, computing the > fake-numa nodes (fake=N) size as the ratio of all > available/non-reserved memory can cause the inability to > create N/2 fake-numa nodes in the physical node. I'm not sure I understand the problem you are trying to solve. Can you provide more specific example? > Signed-off-by: Bruno Faccini > --- > mm/numa_emulation.c | 66 ++++++++++++++++++++++++++------------------- > 1 file changed, 39 insertions(+), 27 deletions(-) > > diff --git a/mm/numa_emulation.c b/mm/numa_emulation.c > index 031fb9961bf7..0c72c85cfc10 100644 > --- a/mm/numa_emulation.c > +++ b/mm/numa_emulation.c > @@ -77,20 +77,19 @@ static int __init emu_setup_memblk(struct numa_meminfo *ei, > } > > /* > - * Sets up nr_nodes fake nodes interleaved over physical nodes ranging from addr > - * to max_addr. > + * Sets up nr_nodes fake nodes interleaved over all physical nodes > * > * Returns zero on success or negative on error. > */ > static int __init split_nodes_interleave(struct numa_meminfo *ei, > struct numa_meminfo *pi, > - u64 addr, u64 max_addr, int nr_nodes) > + int nr_nodes) > { > nodemask_t physnode_mask = numa_nodes_parsed; > - u64 size; > - int big; > - int nid = 0; > - int i, ret; > + int nid = 0, physnodes_with_mem = 0; > + int i, ret, phys_blk; > + static u64 sizes[MAX_NUMNODES] __initdata; > + static int bigs[MAX_NUMNODES] __initdata; > > if (nr_nodes <= 0) > return -1; > @@ -100,25 +99,41 @@ static int __init split_nodes_interleave(struct numa_meminfo *ei, > nr_nodes = MAX_NUMNODES; > } > > - /* > - * Calculate target node size. x86_32 freaks on __udivdi3() so do > - * the division in ulong number of pages and convert back. > - */ > - size = max_addr - addr - mem_hole_size(addr, max_addr); > - size = PFN_PHYS((unsigned long)(size >> PAGE_SHIFT) / nr_nodes); > + /* count physical nodes with memory */ > + for_each_node_mask(i, physnode_mask) { > + phys_blk = emu_find_memblk_by_nid(i, pi); > + if (phys_blk < 0) > + continue; > + physnodes_with_mem++; > + } > > /* > - * Calculate the number of big nodes that can be allocated as a result > - * of consolidating the remainder. > + * Calculate target fake nodes sizes for each physical node with memory. > + * x86_32 freaks on __udivdi3() so do the division in ulong number of > + * pages and convert back. > */ > - big = ((size & ~FAKE_NODE_MIN_HASH_MASK) * nr_nodes) / > - FAKE_NODE_MIN_SIZE; > + for_each_node_mask(i, physnode_mask) { > + phys_blk = emu_find_memblk_by_nid(i, pi); > + if (phys_blk < 0) > + continue; > > - size &= FAKE_NODE_MIN_HASH_MASK; > - if (!size) { > - pr_err("Not enough memory for each node. " > - "NUMA emulation disabled.\n"); > - return -1; > + sizes[i] = pi->blk[phys_blk].end - pi->blk[phys_blk].start - > + mem_hole_size(pi->blk[phys_blk].start, pi->blk[phys_blk].end); > + sizes[i] = PFN_PHYS((unsigned long)(sizes[i] >> PAGE_SHIFT) / > + nr_nodes * physnodes_with_mem); > + > + /* > + * Calculate the number of big nodes that can be allocated as a result > + * of consolidating the remainder. > + */ > + bigs[i] = ((sizes[i] & ~FAKE_NODE_MIN_HASH_MASK) * nr_nodes) / physnodes_with_mem / > + FAKE_NODE_MIN_SIZE; > + sizes[i] &= FAKE_NODE_MIN_HASH_MASK; > + if (!sizes[i]) { > + pr_err("Not enough memory for each node inside physical numa node %d. NUMA emulation disabled.\n", > + i); > + return -1; > + } > } > > /* > @@ -138,16 +150,16 @@ static int __init split_nodes_interleave(struct numa_meminfo *ei, > } > start = pi->blk[phys_blk].start; > limit = pi->blk[phys_blk].end; > - end = start + size; > + end = start + sizes[i]; > > - if (nid < big) > + if (nid < bigs[i]) > end += FAKE_NODE_MIN_SIZE; > > /* > * Continue to add memory to this fake node if its > * non-reserved memory is less than the per-node size. > */ > - while (end - start - mem_hole_size(start, end) < size) { > + while (end - start - mem_hole_size(start, end) < sizes[i]) { > end += FAKE_NODE_MIN_SIZE; > if (end > limit) { > end = limit; > @@ -169,7 +181,7 @@ static int __init split_nodes_interleave(struct numa_meminfo *ei, > * next node, this one must extend to the end of the > * physical node. > */ > - if (limit - end - mem_hole_size(end, limit) < size) > + if (limit - end - mem_hole_size(end, limit) < sizes[i]) > end = limit; > > ret = emu_setup_memblk(ei, pi, nid++ % nr_nodes, > @@ -432,7 +444,7 @@ void __init numa_emulation(struct numa_meminfo *numa_meminfo, int numa_dist_cnt) > unsigned long n; > > n = simple_strtoul(emu_cmdline, &emu_cmdline, 0); > - ret = split_nodes_interleave(&ei, &pi, 0, max_addr, n); > + ret = split_nodes_interleave(&ei, &pi, n); > } > if (*emu_cmdline == ':') > emu_cmdline++; > -- > 2.34.1 > -- Sincerely yours, Mike.