From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E9E1C54788 for ; Thu, 22 Feb 2024 14:04:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F5656B0082; Thu, 22 Feb 2024 09:04:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 880E96B0083; Thu, 22 Feb 2024 09:04:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6ABA96B0085; Thu, 22 Feb 2024 09:04:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5941B6B0082 for ; Thu, 22 Feb 2024 09:04:56 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2D3D0140861 for ; Thu, 22 Feb 2024 14:04:56 +0000 (UTC) X-FDA: 81819610992.10.01C9BC2 Received: from out-181.mta0.migadu.com (out-181.mta0.migadu.com [91.218.175.181]) by imf27.hostedemail.com (Postfix) with ESMTP id B779D4003C for ; Thu, 22 Feb 2024 14:04:51 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ZZ38NdKG; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf27.hostedemail.com: domain of gang.li@linux.dev designates 91.218.175.181 as permitted sender) smtp.mailfrom=gang.li@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708610691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GFm+BsVb+EJLE4T/S60F9nhII3K5Okpm0T3KX+Ypj6g=; b=vsIywClyeFLtaJUUYKnqvEE0OmE0A/1Zb9u7s3AU03ru3jzZtTO7An+3ohMYLbyWm7dmwj rek4NERI0RP6/hVjulo8Pe9/hTD/iR0XrVht9cPQfXqn5wAhj5GwEwKNyO+2UKrVF31tGe 0hQbO3DLRJ44sAY+AXNOP7fkKzt2/hI= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ZZ38NdKG; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf27.hostedemail.com: domain of gang.li@linux.dev designates 91.218.175.181 as permitted sender) smtp.mailfrom=gang.li@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708610691; a=rsa-sha256; cv=none; b=73pTFl1Y3ufg6B0uJ5yyqxE6mBz35vfuI0B43lXoFYYE60y1mitdYmEEWODi5rrsrfT4sW Io7FDau+apTfvGOrff+nric5Www479o4P2aInYFEY/edufskohxIqAUBMmJyvwbcw9EaaK 8uhNwHQOtY3R6Wdr+e3A6Ei+6EdAGcQ= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1708610690; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GFm+BsVb+EJLE4T/S60F9nhII3K5Okpm0T3KX+Ypj6g=; b=ZZ38NdKGJjXCAsgG44/Qo//cf/3cc1EknNNpxFahUoIgcacNCf55oBecrLxSCFmnIx/6Yc zc0joimqN7Kl9zD1PYmr5FlyOXa9qomyj3kBYRaaA7C375f+IuMQnFTecYf2S6/jKeHeH3 JIJykJuzg6dwPf8qafigyzC4+gneD6Q= From: Gang Li To: Andrew Morton Cc: David Hildenbrand , David Rientjes , Muchun Song , Tim Chen , Steffen Klassert , Daniel Jordan , Jane Chu , "Paul E . McKenney" , Randy Dunlap , linux-mm@kvack.org, linux-kernel@vger.kernel.org, ligang.bdlg@bytedance.com, Gang Li Subject: [PATCH v6 2/8] hugetlb: split hugetlb_hstate_alloc_pages Date: Thu, 22 Feb 2024 22:04:15 +0800 Message-Id: <20240222140422.393911-3-gang.li@linux.dev> In-Reply-To: <20240222140422.393911-1-gang.li@linux.dev> References: <20240222140422.393911-1-gang.li@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: B779D4003C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: hck8w3t4dqc57ryoe98txfbenwqa9sze X-HE-Tag: 1708610691-531244 X-HE-Meta: U2FsdGVkX19mIpeKpUS/03TWxYlMistRWJ6ZYMsmjhWiwO6gEi7Yqb2+nFI11h+WTgwzUlMkqDHR2nSfyaNg1npuSg0E9PjcQu/K4klTlxkxAAyptHB4yvGNHtc97Xdx5TSFOEhotN+V8kivKJY3psWcXg2zi0h5/R1BpQUBuqWZk4AStSZJo/5/bf/alDCe1v/QgPfnDDAPtgQvU56ZR71JvyCk0IDjpLaoKDcL+IRx7vlCpChfAGfFvVoKPeKFkLhX8NmI6dlqerwAHgWBVQu0N1ucEUY2aaw+pB5ft8ph7FoEr+/JZv/iKiZJ/p5rW88oI4l6RUvFqYkKBddeIETx0LcEb9q2Ol0y0p5zS1AZna5VWl0jaxiKXRGy4rd5rhRZH7rT47RpI7Yq1ROqpfG3WebiMltD8Ar9XXBXzE+v78aTcux1UuIHqqu39CoPNxRzgHhqkTiOIUrB+qys8/ngio1R4JRNA46xrERddkzW3nH5n8jQwXeMLpgHCN1ht0wEQa+bE1WxAaRNpurDbReOVzrZ30N3aQNtsvFEyi6Y9oHF45WmXq+yBVAbh42JhrNekts79E7s0MAAiumwDUHdscDvUlI1/W2P0TiJitAXfkieWXEpaldDKgqu+OS5da3/Z0PvkU1lXV+XqIOvDJyM3fXKTbhS4DVKHwudDZAcoIzWEeXO8MaD7y+ETBiD8SXx50pm5DOXAOeliX76mBLm62CNg2VnKry5kQe3SmQ7+PMdylrES+EXV/cVIgds/s4jWHdyErnrztcjue4rmhRPIieYKc94aYmBlKMu+ucTRQfY98P5mIYBwLUYMCOa9hmyX9Wlz4Vtv398rs/Q/k8/LbiNWB2uAaqcl7At87agpnBcf8t5Y6CvpuNVl0tifaHO3mfBGzu5UcxH/ykJAf2a5kK+hMyfsDgNtuxXHolAXBWwql2zNKEpJkLW7Hs/FbUmsA1a8niZv04JsQN PkgSKOZh xDsHotOkg9IuX0H4xiblSbW0/djC07oj/aIx9OG4i+iyWu3D7YRFkbMIGqzZirhz2kZJVNk8yf8TB1/SLgXfDFOBB/n7CezGaR2+uJgUN7BAdzX2Ch0WT1PzxxCw4uI9kXqp3PBEfkNZhpih6jdsaHqREYOExmB+gxPIMUF4hSrm1qDKBuwCe9su0zb4+4sAG5h9+eGa5y61JQK8iPNx4miRPMg2TQ/SDxDVz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: 1G and 2M huge pages have different allocation and initialization logic, which leads to subtle differences in parallelization. Therefore, it is appropriate to split hugetlb_hstate_alloc_pages into gigantic and non-gigantic. This patch has no functional changes. Signed-off-by: Gang Li Tested-by: David Rientjes Reviewed-by: Tim Chen Reviewed-by: Muchun Song --- mm/hugetlb.c | 87 ++++++++++++++++++++++++++-------------------------- 1 file changed, 43 insertions(+), 44 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 794f3e6a19bb6..647e52596e2da 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3509,6 +3509,43 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, } } +static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) +{ + unsigned long i; + + for (i = 0; i < h->max_huge_pages; ++i) { + if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) + break; + cond_resched(); + } + + return i; +} + +static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) +{ + unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); + nodemask_t node_alloc_noretry; + + /* Bit mask controlling how hard we retry per-node allocations.*/ + nodes_clear(node_alloc_noretry); + + for (i = 0; i < h->max_huge_pages; ++i) { + folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + &node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + cond_resched(); + } + + prep_and_add_allocated_folios(h, &folio_list); + + return i; +} + /* * NOTE: this routine is called in different contexts for gigantic and * non-gigantic pages. @@ -3522,10 +3559,7 @@ static void __init hugetlb_hstate_alloc_pages_errcheck(unsigned long allocated, */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { - unsigned long i; - struct folio *folio; - LIST_HEAD(folio_list); - nodemask_t *node_alloc_noretry; + unsigned long allocated; /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3538,47 +3572,12 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) return; /* below will do all node balanced alloc */ - if (!hstate_is_gigantic(h)) { - /* - * Bit mask controlling how hard we retry per-node allocations. - * Ignore errors as lower level routines can deal with - * node_alloc_noretry == NULL. If this kmalloc fails at boot - * time, we are likely in bigger trouble. - */ - node_alloc_noretry = kmalloc(sizeof(*node_alloc_noretry), - GFP_KERNEL); - } else { - /* allocations done at boot time */ - node_alloc_noretry = NULL; - } - - /* bit mask controlling how hard we retry per-node allocations */ - if (node_alloc_noretry) - nodes_clear(*node_alloc_noretry); - - for (i = 0; i < h->max_huge_pages; ++i) { - if (hstate_is_gigantic(h)) { - /* - * gigantic pages not added to list as they are not - * added to pools now. - */ - if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) - break; - } else { - folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], - node_alloc_noretry); - if (!folio) - break; - list_add(&folio->lru, &folio_list); - } - cond_resched(); - } - - /* list will be empty if hstate_is_gigantic */ - prep_and_add_allocated_folios(h, &folio_list); + if (hstate_is_gigantic(h)) + allocated = hugetlb_gigantic_pages_alloc_boot(h); + else + allocated = hugetlb_pages_alloc_boot(h); - hugetlb_hstate_alloc_pages_errcheck(i, h); - kfree(node_alloc_noretry); + hugetlb_hstate_alloc_pages_errcheck(allocated, h); } static void __init hugetlb_init_hstates(void) -- 2.20.1