From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CD50CA5537 for ; Wed, 13 Sep 2023 10:54:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4D386B017A; Wed, 13 Sep 2023 06:54:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 79C8A6B0179; Wed, 13 Sep 2023 06:54:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5EF526B017C; Wed, 13 Sep 2023 06:54:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 308416B0179 for ; Wed, 13 Sep 2023 06:54:11 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0236E160B54 for ; Wed, 13 Sep 2023 10:54:10 +0000 (UTC) X-FDA: 81231264702.17.CAB852C Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by imf23.hostedemail.com (Postfix) with ESMTP id EC0B214001B for ; Wed, 13 Sep 2023 10:54:07 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Jv37r9gL; spf=pass (imf23.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694602449; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=M/jwYmO43zAy8dw1dylGWTby1uHmPD0/fuY9NXytdPo=; b=kDf4E8mWF1IWN8KLwd0PEVQ7p9nmxQnAzbiWk3fRKBFKNNAKfLKOm7USZv/8dZ7gP5YMmY jP7IDd0fKBYwoyfPqJXXcgSltQzWgWxBGlq2+gdNKlvRQB05K5G2zF/jsH1ooOCltGxg/m dYkj/AJOhjUhs/pTjygJL+0z0oKgFiY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Jv37r9gL; spf=pass (imf23.hostedemail.com: domain of usama.arif@bytedance.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=usama.arif@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694602449; a=rsa-sha256; cv=none; b=wtENIMpQZFJ9hqcQdCVMNpFHGg4FVh4QIyWIb59V5EFvJXdXKzo0gfdXlmw1X372pLllk1 YbnjVnPX74MOx8L3nWpm3rnARsQh1/Z1x9oGXUC7ejvKZEyF1z4HMpp+sWXGUWG3mf/6Uy eULrea0AoejCpXmCycBZTkekUqCYviU= Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-403012f27e1so47548065e9.1 for ; Wed, 13 Sep 2023 03:54:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1694602446; x=1695207246; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=M/jwYmO43zAy8dw1dylGWTby1uHmPD0/fuY9NXytdPo=; b=Jv37r9gLWdr73PDX0xneQaUT/2feUYL1NrEteuy8kSuIQ1xoopJozbF3yO0nghjaOM S4ne8SeC/T7hvNa04fV55zKKU+GRackaDeUJUeFB4g1mzsQOVxOb5cq1jCxhpdt1h7kT Tzga3Ljs1ESTMup41x0IprW5qz9pj/pQmxZnIEtJYh+BAwPLMIqtPuEPOZraJ8e8NqNE VvENhF0yCr3XC0l06EjqFdeuoCwRnVShGKZREdVnmRgwUAvN2tHuVR5JlykhejNdYXVY MeqZBi3GzUko+aNfyrwAdSjmLcuC+CVehNr90GDByN/CB0j7aKiBPdq4O/Bk2JP5YmUD w9mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694602446; x=1695207246; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=M/jwYmO43zAy8dw1dylGWTby1uHmPD0/fuY9NXytdPo=; b=Rxdx/Ex2/gzktRYK8/PR9Wa+FAoAQOcVHNOoqM6XDQagNEB3NBOrI+x77eztpnSb4W F4xw3jtymO7TscV/nEwDzHIi6+gtept5DbUtNfvQZUoRnDC9dz+X4GQZVLaX1E35MJ6P HmAIlr3lMv4g+xFQeRZ4viJA72oEMyBaaq/5+7n/wgpjqtp3ohI/p502gWJJd7oPmHFu JWpt3/SyDEpqGy2NMphtj49x8jzWwG7+qNPPuFp1OaX9IE77d80x0ukKWF8B1STOcQd0 krUsGxDm8tuxOstmPFn2AKqKfnb9VxYAdVLybm59ujZWuOzp8sfbsuJPIWESIxq/GCN0 PMpQ== X-Gm-Message-State: AOJu0YxyTZfpnWbJLFWSaOIc9w6Yw4RFG3ruHHQrE6VzIwraHGxtWAGb /+nBsuA5Ai2nPuz6WHOv6FnqhqReSS9dIWqeGEU= X-Google-Smtp-Source: AGHT+IG3TgIhnp1ot645v42otah+o1cULY/OJnOLImpb0/Lvb/J5aq/M+qX9WvdUTwm15PjcuRioHA== X-Received: by 2002:a05:600c:2144:b0:3fe:5501:d284 with SMTP id v4-20020a05600c214400b003fe5501d284mr1760853wml.11.1694602445408; Wed, 13 Sep 2023 03:54:05 -0700 (PDT) Received: from localhost.localdomain ([2a02:6b6a:b5c7:0:59d5:28b9:7f13:cd1b]) by smtp.gmail.com with ESMTPSA id p36-20020a05600c1da400b003ff013a4fd9sm1757055wms.7.2023.09.13.03.54.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 03:54:04 -0700 (PDT) From: Usama Arif To: linux-mm@kvack.org, muchun.song@linux.dev, mike.kravetz@oracle.com, rppt@kernel.org Cc: linux-kernel@vger.kernel.org, songmuchun@bytedance.com, fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com, Usama Arif Subject: [v5 0/4] mm: hugetlb: Skip initialization of gigantic tail struct pages if freed by HVO Date: Wed, 13 Sep 2023 11:53:57 +0100 Message-Id: <20230913105401.519709-1-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: EC0B214001B X-Rspam-User: X-Stat-Signature: at4md71pxsptmkd7w8o8mfy7x6nirw1g X-Rspamd-Server: rspam01 X-HE-Tag: 1694602447-977127 X-HE-Meta: U2FsdGVkX18kVc9EXyPkrcguFy6DYwe12/zajWKwSkincDK9LoXXYeevjUcBOQmZByFn/60+WlfdxdLsXc/dI1JhnYLHNeAL24U2DyT7MaugCPuuWSd4tALVFUddTXMqUtlmb6npVq/1P6NY/HiPx0jbuaw28MMFM8nGdjKz70M4uIOOsHtK6SAh73zGgRiHXmIdUQwZaVN19We5pOcQ5qiA0eJANPge7YIu3lVz92XJ7q7FXt2pp+tHv4RF8bmC+j8T2C5lqxtiAsvmOramNx+eK9mJz+2OhCthJQWB3pfOd0NDmVpC88WCLzMu1i2tOub2pO/y23mrNsxHNIDTz/8pj6ayxEntUGPVmxLnZogyuEZzWA9U0jGoBDQmkT9CvO2S8/neiD6I65bpnJnQ3hwLQgxzOJsZeMcibaUtOboe1gcdM1NIOTxsJVseN61wV+La6i/JmM24wfW910RHxy2+PpD0OJDBMCSUcHdRGUoxy8N1TURyKD5JNU8zH+x+bgoRgkeoPbVq+uMGqsgwRBPYL0WmAisCXamUdiNYY2I5h6ETnArWm8LJJLHfSJrCZgaJc3flqZNrJhzfy43Hb3KI6oUUdIJFjCC1p5cXVQrh77KHwWjKwmvTquTOquO51Y/d5wS1TJGJxK8VgOcbNO/W/6aGA0DeMWGmc/rAzaA7cLn3UFzq6SjZeBxrYQO5h4kdPNYcVGMcDUGtSKnl5twbeKfZ5BIzzdJGTp2KX6Y/mAuSZQOI0VlQVcGxb1vzslvCE49PI65E/7vbTeCGeo0JeSERQSDSvRf+vDzgN768u+POgDDQS5SiUM3erAFsvo/RQ3+fVOaI/4bYkQrfH7BOanBFZ9C5d30inp/xPDHejEufEzVo4xeMOHyjcM8ccH38vuPgfPjDwZWzaRz+WCxzLeTUoEm1WJ3FRoSekXkXX6PH2FUy571C82BWfKso9Ro+lvf3qYB6hLaxw0Q Ece+Uf6a LRYkorRSqjIFJyZNSeIZDPV2i6l4gfD5W7kIR1f1+nxBBXBuc+pgJ5RerfV33XOS1DfJ4l5kNHe6awmLkQidvxJhkfA4k/6caIuLlXsjX21SxQk1ztjmz8dtC3VSl7f2+oN+dZPr8Ouz6DL1KUBl4wZXxsCXic9OKDDmvzgT1698EnPRXvyzKbNUGl632Teqj+SmVYmnH6tuj3q3dpNBFgfG+dvhSmQSTyMYwZ4/2mvVN5BCw9aP2ihnkjs88DsPiLMwZGKH7+ZqpfFhDlW8wq5bRkpW/oZDEJ41nAh98b04vzGXAJz+Yd75OBNsQIniHaBD9/gXEdPXMT1+sVVae5ZNB4Y4PREFZEHPUyrrw9FPHbMU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This series moves the boot time initialization of tail struct pages of a gigantic page to later on in the boot. Only the HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page) - 1 tail struct pages are initialized at the start. If HVO is successful, then no more tail struct pages need to be initialized. For a 1G hugepage, this series avoid initialization of 262144 - 63 = 262081 struct pages per hugepage. When tested on a 512G system (allocating 500 1G hugepages), the kexec-boot times with DEFERRED_STRUCT_PAGE_INIT enabled are: - with patches, HVO enabled: 1.32 seconds - with patches, HVO disabled: 2.15 seconds - without patches, HVO enabled: 3.90 seconds - without patches, HVO disabled: 3.58 seconds This represents an approximately 70% reduction in boot time and will significantly reduce server downtime when using a large number of gigantic pages. This series is based on top of the patch "hugetlb: set hugetlb page flag before optimizing vmemmap" Thanks, Usama [v4->v5]: - (Mike Kravetz) For tail struct pages, change from using set_page_count to page_ref_freeze + VM_BUG_ON if ref_count != 1. [v3->v4]: - rebase ontop of patch "hugetlb: set hugetlb page flag before optimizing vmemmap". - freeze head struct page ref count. - Change order of operations to initialize head struct page -> initialize the necessary tail struct pages -> attempt HVO -> initialize the rest of the tail struct pages if HVO fails. - (Mike Rapoport and Muchun Song) remove "_vmemmap" suffix from memblock reserve noinit flags anf functions. [v2->v3]: - (Muchun Song) skip prep of struct pages backing gigantic hugepages at boot time only. - (Muchun Song) move initialization of tail struct pages to after HVO is attempted. [v1->v2]: - (Mike Rapoport) Code quality improvements (function names, arguments, comments). [RFC->v1]: - (Mike Rapoport) Change from passing hugepage_size in memblock_alloc_try_nid_raw for skipping struct page initialization to using MEMBLOCK_RSRV_NOINIT flag Usama Arif (4): mm: hugetlb_vmemmap: Use nid of the head page to reallocate it memblock: pass memblock_type to memblock_setclr_flag memblock: introduce MEMBLOCK_RSRV_NOINIT flag mm: hugetlb: Skip initialization of gigantic tail struct pages if freed by HVO include/linux/memblock.h | 9 ++++++ mm/hugetlb.c | 63 ++++++++++++++++++++++++++++++++++------ mm/hugetlb_vmemmap.c | 4 +-- mm/hugetlb_vmemmap.h | 9 +++--- mm/internal.h | 3 ++ mm/memblock.c | 48 +++++++++++++++++++++--------- mm/mm_init.c | 2 +- 7 files changed, 109 insertions(+), 29 deletions(-) -- 2.25.1