From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8255EB64DD for ; Thu, 27 Jul 2023 20:46:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C7FC6B007E; Thu, 27 Jul 2023 16:46:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 951EB6B0080; Thu, 27 Jul 2023 16:46:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CA886B0081; Thu, 27 Jul 2023 16:46:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6B6BF6B007E for ; Thu, 27 Jul 2023 16:46:40 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 426E81201A2 for ; Thu, 27 Jul 2023 20:46:40 +0000 (UTC) X-FDA: 81058575360.11.B7CFF38 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by imf07.hostedemail.com (Postfix) with ESMTP id 6193B4000F for ; Thu, 27 Jul 2023 20:46:38 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MHW+96et; spf=pass (imf07.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690490798; a=rsa-sha256; cv=none; b=GmIkbmHiUe/+lXecf3cGRgpL5lshLy1xmlPvOiOfRUTGXI2wiowU/ULJ3DyxA9+7LvgymS NwkuoTpvEj6RamThFag6XN4jjfdmVdPt57GPx6kttlkF0WDCzbWLKnOUJkwacX9eL7Ytne N8EN/5VCbrqRLCNMWpPi8BPBXvFEti4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=MHW+96et; spf=pass (imf07.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690490798; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; b=W29hwWzXCgxffehWhgYsSb3UgRhrUJ1zlsiv5DQMDVwPvovl2aLL0D9VR9oOMTq3u6ppZ4 VuVAXgkwzvLeTZxfbhGhKs2gwFdrxDMkfNPi7cDtrjJVBCEZy2qMrQtx0R9YEukTSvyCfm FYqF8qRWTUilg9y3Vny2eDPm3sgMExg= Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-31427ddd3fbso1440879f8f.0 for ; Thu, 27 Jul 2023 13:46:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1690490797; x=1691095597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; b=MHW+96etwxTrHcysNxNv4Jt5Keu/21p5ZeNuDIbQx9GnIkEA1kq7sTTqw6Q/8BayM4 1Acht/ZPiVv8tOChy+mgBTfHG6ixCBKorlj2dJl7YTk0/sTWgbMWE9WJCFXy+OQ9Iijl F797zrYkH1WHN8+kZpWGtO4V6mpRCt3ijeyEEL/x0uHMVkx2JVs2E0JF144vy0BqQwCm brRLxremJQxfBWLsYq+wA+OhATOq4v3hK/HHE03k794S7TLjgAEkUPQNfZuGNCFBakRv 3NI8Jepy+Nx2mW0XqXn6WjNkB/tP3+1AiL7EGltSxqblDgE7Db3FfhwIHlnqEs7zbPpH c/9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690490797; x=1691095597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ckgW/P+ejqydrp6JQqQrrU5HjkGQNsndgwMWGYDB/pc=; b=YDJuMot4JCrGxGZGg9u7f5F2mkB5654ATH5ID1FJScWviuMd0gbih1I6OOz3bNhjKg e4gg4DWP+GLN7CMgC2nP0WJ0nCBmfEvYgtbIUqcLc5sJ2NX3EmBYAOx3ZNRVTZg/uGH9 0RbDTdFee056HMPNuTV0N5jX1kf+6BmZ/hA5DRBD9MSsUhpd1PoYd/jRB4X0z0MNmg4q S9OGL9ZeqNHXFZvSqtcO3heaNnf0ZKipeY7Q8Q9MYOGEo3sJtM0S/KzEqrYIZKynTK28 G9RdpwuNuPAnFmqH5ZGQRPSziIaMASuZvxrAXUsOrlyAbEfbmZgDFF/UFNdKzOskBIDa nu9Q== X-Gm-Message-State: ABy/qLbzMXXYujy6ZAl7JCIBhqSqcu5l6oJjH0Jo2Z84mrD4CLrBDh1K CpJTM+g5as12fmx0M82HeefjPCjIhiSV2SFFQoA= X-Google-Smtp-Source: APBJJlEO7+YO7D/m+lNz0Wmsb1Fis9T5rZ6fiE8hYl22nZAClbV/pTs8BoCD41J0AwcBpujlXdeyTA== X-Received: by 2002:adf:db44:0:b0:314:34dd:aaec with SMTP id f4-20020adfdb44000000b0031434ddaaecmr229166wrj.8.1690490796779; Thu, 27 Jul 2023 13:46:36 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b465:0:7e3e:db9e:70fa:9ccb]) by smtp.gmail.com with ESMTPSA id k1-20020a7bc401000000b003fc06169abdsm2701400wmi.2.2023.07.27.13.46.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 13:46:36 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, fam.zheng@bytedance.com, liangma@liangbit.com, simon.evans@bytedance.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v1 6/6] mm: hugetlb: Skip initialization of struct pages freed later by HVO Date: Thu, 27 Jul 2023 21:46:24 +0100 Message-Id: <20230727204624.1942372-7-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230727204624.1942372-1-usama.arif@bytedance.com> References: <20230727204624.1942372-1-usama.arif@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6193B4000F X-Stat-Signature: wb1f8918ro4ub3pugta3yzpgk7maystb X-Rspam-User: X-HE-Tag: 1690490798-696374 X-HE-Meta: U2FsdGVkX1+Z6LBYLgT7x72yXyYtJ11O79v+95PUAToll6GW6/W7De1EOAPmHf/2naNc0Oe+3IdMwa51MSbPs191rSh7pnqCGrGTIhZ6BmO6MgFp33XssOmOaHs43YZgkIEpwEeHxRIyyGmcwOCR8Bnug07KrPA+Ue5miI+ASlLrjkPtBx4PnyCBDUrT2HQqpats8fosM/3dDX+9QqpAM19sHe4v+2QtS7sSBTkFtNpjAFYMGKGIASFod4b5hxNUUDKHYnCztLoP5og79NOql/0BVb56AGZtkXW02MNldiMjgD6X0jboKrpqCDwPug/vcm+Jb4jo96AY/Pod1UDu/hx/ouq/uJUHcZN6rN9aNETTy8KAXBrbSVZ9cRNxYrN89tMl8r5hDjYBJj78PHmjLyMCsfkTk82+KAMscbCJ4vzFBzznBTedJc12WL6IDCWBPJKOO2w59iQCzIOs0htn6K7T1fg4TDFzX93KvptrB15KxZ974aj89GM8GEgAPCUIL7XBGYzL8R+mPOfy6Oaf3gsUPj6M/RlY+oolWf/rvoMMHA3QF6lVvW7G4Q1b1nM7j3EyxMMcJxeA/t/csxHlyRWagEVhkN2Hm6A9YiUTf0xuiWh7TrW29U8HAIrMkGchEu2i6+8WVHgeUqxDJWHewQwPpoPIK656GT0sz/ILWx74qWhu5RMzDFq3gV2N+687pZz1DTIUMUm06qfNLam0ya/aul6NQjmLFCS+JHE1ywxYvQu4v5PiCIzIkr6hwYHmLv7rn50w1v2/TKpvJWpfYMfEaBDryzyQzyFMN5LNoPWFcMZUXJWdLabXIdfw5YBCJXg7uicxLZQ/Dejb/FVvQrCTTwukPUX/vlRXWAdPvldktIjWv7RThI1pD/StW0qiz22x2m0eaJyRdaR/4Tk94Aiislvd3KoFaSsPmWk5AUSn2uDG4qWw2lYUUMAOBK3q90NPKid9WKonLAoDCk4 B7ZhxkC7 GL8yDy700JcdMsRKEzXBk2xW15No4N1CiZ9apB6AYOuqsJKnXLayUv77DwVpqkggDkEdAnvKNLpfgSb6+JpMxusURDNHxvWqqXMqSiM4eacBZKvVEzeRO36sWsU1wn9xn5kW9TrCI3jdIzKbSYFAj/v8RFZaUiXmUIYcD0kAHQeJd+1NxUm4oFcHwTClA0hGRVK2JkwSRr50cfmjgeN0GjzdLjN2OjlhKgwS6amUuV1XknbqK1WEPRA7yFZh5xRJfI7E+zzL52yxlPRveQs3qFTmsM14AN5VIsWmJ/74cSrnfLmhFO+IrrhfN2+S5dUsFjiazXxx1qB9MSp1fdL7JIeVuflRnd1v/JYTKoyI6Y2Tm+Z7i1ZA8Y1FQ3QG3Ntz/egTwgkc9FRVOwa/6UBzy/deVqD6LuCxE2X628eGezIxcLNw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is done by marking the region for which to skip initialization with the MEMBLOCK_RSRV_NOINIT flag. If the region is for hugepages and if HVO is enabled, then those struct pages which will be freed later don't need to be initialized. This can save significant time when a large number of hugepages are allocated at boot time. HUGETLB_VMEMMAP_RESERVE_SIZE struct pages at the start of hugepage still need to be initialized. Signed-off-by: Usama Arif --- mm/hugetlb.c | 21 +++++++++++++++++++++ mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 3 +++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c1fcf2af591a..bb2b12f41026 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3166,6 +3166,9 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) { struct huge_bootmem_page *m = NULL; /* initialize for clang */ int nr_nodes, node; + phys_addr_t hugetlb_vmemmap_reserve_size = + HUGETLB_VMEMMAP_RESERVE_SIZE * sizeof(struct page); + phys_addr_t noinit_base; /* do node specific alloc */ if (nid != NUMA_NO_NODE) { @@ -3173,6 +3176,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_rsrv_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } /* allocate from next node when distributing huge pages */ @@ -3187,6 +3199,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ if (!m) return 0; + + if (vmemmap_optimize_enabled && hugetlb_vmemmap_optimizable(h)) { + noinit_base = virt_to_phys( + (void *)((phys_addr_t) m + hugetlb_vmemmap_reserve_size)); + memblock_rsrv_mark_noinit( + noinit_base, + huge_page_size(h) - hugetlb_vmemmap_reserve_size); + } + goto found; } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index bdf750a4786b..b5b7834e0f42 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -443,7 +443,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); -static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); +bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); /** diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 07555d2dc0cb..cb5171abe683 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -64,4 +64,7 @@ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) { return hugetlb_vmemmap_optimizable_size(h) != 0; } + +extern bool vmemmap_optimize_enabled; + #endif /* _LINUX_HUGETLB_VMEMMAP_H */ -- 2.25.1