From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADA8DEB8FA5 for ; Wed, 6 Sep 2023 11:26:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9140280015; Wed, 6 Sep 2023 07:26:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E0C26280013; Wed, 6 Sep 2023 07:26:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3677280015; Wed, 6 Sep 2023 07:26:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B4BEB280013 for ; Wed, 6 Sep 2023 07:26:20 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9274EA0856 for ; Wed, 6 Sep 2023 11:26:20 +0000 (UTC) X-FDA: 81205944120.23.DD28765 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by imf09.hostedemail.com (Postfix) with ESMTP id AEFE9140009 for ; Wed, 6 Sep 2023 11:26:18 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Zrwhokib; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf09.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.221.52 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693999578; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FStpXjXfy05Wptcs5VKtJaC3uUKwiP44RVFSTPHctoE=; b=3GLwuN9aFFL9r8YsyoYC8h9JvMp1qBqmZh1GTc9ds6pp/7DDvkm0ez+N6BADKtzGQs+LjA S/2pZbeeTUTagmDnVOsrVQpR5/S1jk+y+ptT46CAbNYYaZNIsY4LGuldvsXKs9W/rdUHiE wna46tuz7XuRTehE47q76LBE+NuDH84= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Zrwhokib; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf09.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.221.52 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693999578; a=rsa-sha256; cv=none; b=eUy2CrsLV/JoJqZWWrOWEn91SXKfDJBPQySZgfnm8ryAUX1hcFy4PNM6476ah4Rm6XT8EM Mx4/druD9Zrx0xrvfwZWsvxigad8QGPtxLBhhisvEbuy/Cl2uRAu4SkKUA1AyKBFHw3irT qKO3MVKxriyAWqmi8XAoc66SkgB+Nng= Received: by mail-wr1-f52.google.com with SMTP id ffacd0b85a97d-31c615eb6feso2978242f8f.3 for ; Wed, 06 Sep 2023 04:26:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1693999577; x=1694604377; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FStpXjXfy05Wptcs5VKtJaC3uUKwiP44RVFSTPHctoE=; b=Zrwhokib+cSBwgLbqcLxtmKd/7jnp9KG/Naaln+fX+qWl6qSTfV/Ogcxq2pV7XPr7c HTKo8zpoPru1L3ZLw9LpQ59W4/Cy4p2F0R1MlWW26kDDC6R8wlmiCk3FMj5W9OEuXyjm 4xNUrks87h17ebGCQJcAMHK61TEqciCnWc2h1s5gGcf6/6uUJCVc3XdQMD0ILG2klwge sLlnDQBr7s51BCJpkmVZpZcAJMNsOyS26unS65RaPBjb+Y/7aHWz6N27tYYTWuOv+R+L MKZE3CFKxYRQqeqSO/e55ZiVFG4HZ84GVLchj1zL+ntv/PIX7Vag/T2/Y1QFS84TXz9Q TS9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693999577; x=1694604377; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FStpXjXfy05Wptcs5VKtJaC3uUKwiP44RVFSTPHctoE=; b=HGtEMb5KkvCEzgmegotIvWznf7ZpwwwupBBOdldTpH9AvD4AmZoozoW247IW8qrMqp /k9S+dRqcQMgCB0aWrVPRQb3rKPK8wp0CN2D2gizwYCKizcfm0sPMuoX+rofncSJw+rr LKW0oh3t6ovNk7KqRPdq2T4ZCmkoDSDSxDnwZL6f7BDXz+VDXImmhtpOlm1VqaQ8XknR HNA/78CwdoJOk2CyK7HZD1KC38mp+wTeyxaet40SMKwCvF3V66tKFTqlLjc8sgCxqq3b jO5PL/Ej1F9yDI2rPala6XzrLOKH6i+GvraA9CigCpgdf/CyS31dBgGiYdCrpcLTc0JU o16g== X-Gm-Message-State: AOJu0Yz4tEwMjoWFWlctAaymjOU87ujMRzs3BrXpcjKl1PetLXYYe8eZ WeXplMpLxENb+P/HuXmEsaPeNKU9lw6/MSqMaSE= X-Google-Smtp-Source: AGHT+IG3C1Ltp43a9GTFE7tcAw7+FslSjp6rOutSzsw4cYuJWx+ebBM3Lt3tEbnjyOzPbJs8SSeFmw== X-Received: by 2002:a05:6000:11ca:b0:306:46c4:d313 with SMTP id i10-20020a05600011ca00b0030646c4d313mr1851070wrx.28.1693999577038; Wed, 06 Sep 2023 04:26:17 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b5c7:0:6dce:31d8:efb1:5d81]) by smtp.gmail.com with ESMTPSA id n9-20020a05600c294900b003feff926fc5sm19517038wmd.17.2023.09.06.04.26.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Sep 2023 04:26:16 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, songmuchun@bytedance.com, fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v4 4/4] mm: hugetlb: Skip initialization of gigantic tail struct pages if freed by HVO Date: Wed, 6 Sep 2023 12:26:05 +0100 Message-Id: <20230906112605.2286994-5-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230906112605.2286994-1-usama.arif@bytedance.com> References: <20230906112605.2286994-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: AEFE9140009 X-Stat-Signature: 7auiowcfcebutw5bkxe4uzbrr3zagem9 X-HE-Tag: 1693999578-323896 X-HE-Meta: U2FsdGVkX1/bwAGOosup581rTKIUsKr8eOX5jTVrumsnidL01E4FBzWWCXq5zz2kKGQSe1G2B/CS/+hA/1aOLezMD4QMgOd8bXh4NObXkkE+DVjBI10Fd8Ms+PFh88N6Pofq4ZzXzqcxk6hB1XNA7zXly4rM8cO5LxviVQHlOPt3giEy8xPBfWl2i0mVIi+/uizYYtooJ06UBza6pjiPo3pc52Aaz2nEPLTkEZDVK3yqF76FjOfTGDFcrRbQIS/LJo4IhCtEOpxN8uDWy3xKxWu3G+BjxmH9ZzDNQLpFfDV4YWpo7u1mL7Mu834JMUyhmDMiL/PTEdzqSIrHFMef66SqOth8aQuEA2nanmSp1TuRKeHn490LbiLT1S5HrO7kluwhNdnxYf5RLpH/TAmsbc0+7jy3nc0sP4Rhx9Gm9bLI5DiUZSi/HUH6GhWnNupjmYL5ori1Z41xwn7gQ/PkoauwLpxc1txym9g480cVnEl2VQOrNFqjv54BGG3v6FX+yXAIecZ34/ybluU1KLLAnYMBqe1up0/gzhRZph8FgFYIiqR9j6rJUZkWx3i6XkM8i42eqpqph/vY5pF21YExhqfykEYhCURPHfSaPeHOkipX3Q4n4mKn+yQssmVK+Jb1aYzTYezQB2z7gwAjvRl/19Kx5dp22g2NKTON90RUTaQoknlieDN6Rx1WKKdVf5a1UC2MdgJThaOo/6CmNM8/Mi5NQFs0FobwrxgkRxXnbJ4m8/vBTLHC4LUzUC9126lsKtWvRotJyZ8qvq+Z21hMRVAOBbHWB3D/ISmhA1Yri7h4HWlU0Rgg2d0Edxu/6MqRmTUgVVL2dUG40yWaICdugnui3brXfkJD3J4OtYGEeoYbKf7O0Z4FjSsyUWrYj4mdpXgxve/+VcwX2kaYM8DKwBsRiiJaBKh/nMA3TnVHMKl75Tg7LnY4jJnfFCyXCBwrJ4owFzDjyV9GOGVYBr7 /bP+62CG IHlvYMZ0I3UoBD43+m9uNqgVPrRx/8UOJdsx5Q8fwFShOVcyWZvj1KFRWnhLiYO37gGRE8WVa3ZdK+MmEIzYSU0hJpje/J0zLWmUDzdAs2Ch77SEom5FD6hmuOXZHuctlxNEIUxzQXe2v8zAygrwiV/ZXZdYpAW6CP6Icoso7H3VUnX/2BPWhX7KnlfCxbBkiOa2eoVZM09RE9nQqQ8UfIkVKbmq8TJ29wbCUfCvHN2ONgZo6CbIph8ibfsAiPmp6PdLcRJB6ACwq7/xepFgT0k8El9u4v2uWzjhsQLCn/pM96G5xzbf1Z/2amDOuDWhA/BGiDeO/hZwIHPqmWLlk65FIAfBKK0lAhEGJTnfCg+uNMriSRMQjR3Rzy9l23ZHKas/AS8B9xHyKMRNS/r+04gp6LxnfeLGw7GiJ/YAPeqrjTa0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The new boot flow when it comes to initialization of gigantic pages is as follows: - At boot time, for a gigantic page during __alloc_bootmem_hugepage, the region after the first struct page is marked as noinit. - This results in only the first struct page to be initialized in reserve_bootmem_region. As the tail struct pages are not initialized at this point, there can be a significant saving in boot time if HVO succeeds later on. - Later on in the boot, the head page is prepped and the first HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page) - 1 tail struct pages are initialized. - HVO is attempted. If it is not successful, then the rest of the tail struct pages are initialized. If it is successful, no more tail struct pages need to be initialized saving significant boot time. Signed-off-by: Usama Arif --- mm/hugetlb.c | 61 +++++++++++++++++++++++++++++++++++++------- mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 9 ++++--- mm/internal.h | 3 +++ mm/mm_init.c | 2 +- 5 files changed, 62 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c32ca241df4b..540e0386514e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3169,6 +3169,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) } found: + + /* + * Only initialize the head struct page in memmap_init_reserved_pages, + * rest of the struct pages will be initialized by the HugeTLB subsystem itself. + * The head struct page is used to get folio information by the HugeTLB + * subsystem like zone id and node id. + */ + memblock_reserved_mark_noinit(virt_to_phys((void *)m + PAGE_SIZE), + huge_page_size(h) - PAGE_SIZE); /* Put them into a private list first because mem_map is not up yet */ INIT_LIST_HEAD(&m->list); list_add(&m->list, &huge_boot_pages); @@ -3176,6 +3185,40 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) return 1; } +/* Initialize [start_page:end_page_number] tail struct pages of a hugepage */ +static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio, + unsigned long start_page_number, + unsigned long end_page_number) +{ + enum zone_type zone = zone_idx(folio_zone(folio)); + int nid = folio_nid(folio); + unsigned long head_pfn = folio_pfn(folio); + unsigned long pfn, end_pfn = head_pfn + end_page_number; + + for (pfn = head_pfn + start_page_number; pfn < end_pfn; pfn++) { + struct page *page = pfn_to_page(pfn); + + __init_single_page(page, pfn, zone, nid); + prep_compound_tail((struct page *)folio, pfn - head_pfn); + set_page_count(page, 0); + } +} + +static void __init hugetlb_folio_init_vmemmap(struct folio *folio, struct hstate *h, + unsigned long nr_pages) +{ + int ret; + + /* Prepare folio head */ + __folio_clear_reserved(folio); + __folio_set_head(folio); + ret = page_ref_freeze(&folio->page, 1); + VM_BUG_ON(!ret); + /* Initialize the necessary tail struct pages */ + hugetlb_folio_init_tail_vmemmap(folio, 1, nr_pages); + prep_compound_head((struct page *)folio, huge_page_order(h)); +} + /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_ORDER) pages. @@ -3186,19 +3229,19 @@ static void __init gather_bootmem_prealloc(void) list_for_each_entry(m, &huge_boot_pages, list) { struct page *page = virt_to_page(m); - struct folio *folio = page_folio(page); + struct folio *folio = (void *)page; struct hstate *h = m->hstate; VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); - if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { - WARN_ON(folio_test_reserved(folio)); - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); - free_huge_folio(folio); /* add to the hugepage allocator */ - } else { - /* VERY unlikely inflated ref count on a tail page */ - free_gigantic_folio(folio, huge_page_order(h)); - } + + hugetlb_folio_init_vmemmap(folio, h, HUGETLB_VMEMMAP_RESERVE_PAGES); + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + /* If HVO fails, initialize all tail struct pages */ + if (!HPageVmemmapOptimized(&folio->page)) + hugetlb_folio_init_tail_vmemmap(folio, HUGETLB_VMEMMAP_RESERVE_PAGES, + pages_per_huge_page(h)); + free_huge_folio(folio); /* add to the hugepage allocator */ /* * We need to restore the 'stolen' pages to totalram_pages diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 3cdb38d87a95..772a877918d7 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -589,7 +589,7 @@ static int __init hugetlb_vmemmap_init(void) const struct hstate *h; /* HUGETLB_VMEMMAP_RESERVE_SIZE should cover all used struct pages */ - BUILD_BUG_ON(__NR_USED_SUBPAGE * sizeof(struct page) > HUGETLB_VMEMMAP_RESERVE_SIZE); + BUILD_BUG_ON(__NR_USED_SUBPAGE > HUGETLB_VMEMMAP_RESERVE_PAGES); for_each_hstate(h) { if (hugetlb_vmemmap_optimizable(h)) { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..4573899855d7 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -10,15 +10,16 @@ #define _LINUX_HUGETLB_VMEMMAP_H #include -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); - /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See * Documentation/vm/vmemmap_dedup.rst. */ #define HUGETLB_VMEMMAP_RESERVE_SIZE PAGE_SIZE +#define HUGETLB_VMEMMAP_RESERVE_PAGES (HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page)) + +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { diff --git a/mm/internal.h b/mm/internal.h index d1d4bf4e63c0..d74061aa6de7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1154,4 +1154,7 @@ struct vma_prepare { struct vm_area_struct *remove; struct vm_area_struct *remove2; }; + +void __meminit __init_single_page(struct page *page, unsigned long pfn, + unsigned long zone, int nid); #endif /* __MM_INTERNAL_H */ diff --git a/mm/mm_init.c b/mm/mm_init.c index 50f2f34745af..fed4370b02e1 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -555,7 +555,7 @@ static void __init find_zone_movable_pfns_for_nodes(void) node_states[N_MEMORY] = saved_node_state; } -static void __meminit __init_single_page(struct page *page, unsigned long pfn, +void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { mm_zero_struct_page(page); -- 2.25.1