From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E346C4332F for ; Tue, 18 Oct 2022 03:22:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E04F26B0072; Mon, 17 Oct 2022 23:22:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D8C566B0075; Mon, 17 Oct 2022 23:22:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB8C86B0078; Mon, 17 Oct 2022 23:22:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A61B56B0072 for ; Mon, 17 Oct 2022 23:22:28 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7EF24A9D2F for ; Tue, 18 Oct 2022 03:22:28 +0000 (UTC) X-FDA: 80032622376.21.5BF0669 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf25.hostedemail.com (Postfix) with ESMTP id 13BF4A0027 for ; Tue, 18 Oct 2022 03:22:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666063347; x=1697599347; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=5mUUeKwIz76PjPvI9DB7Y8jeeiA6yd+TrNYqbiZzHYg=; b=ZcipxD0mTUM5kEvEjRoj+mwtYmCkUHCG8im36+aHIBVSoye1ptSh/8di BOU0h7jE1TiCKm/L5hwYNc4meSB7zW7PFkanyUhAUMrjemRnKei4FoPg9 f5r+/lCk8ZGqx0RtF7Ql0COg2wC/hZ5hrp/P2xEXxAXb62f3WLqcbibms IxORA48D7QUEgrXfobVEFX/kHTaXtPvYIK1lYwWNYZwYHL/aDxaEXnWp6 G1T5xrcrak54PnjP3hK9//GpiE0e7GfDw/H8Fbf1Rkh3j07gnrPjDklGY DSfcr7hflxfGAwlTVvzqMg7CY4O6ApufnmNX+VjTklpUNqXzij7VbEuFC w==; X-IronPort-AV: E=McAfee;i="6500,9779,10503"; a="368012471" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="368012471" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2022 20:22:09 -0700 X-IronPort-AV: E=McAfee;i="6500,9779,10503"; a="771015308" X-IronPort-AV: E=Sophos;i="5.95,193,1661842800"; d="scan'208";a="771015308" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2022 20:22:07 -0700 From: "Huang, Ying" To: Cc: , , , , Subject: Re: [PATCH v3 1/3] mm: add thp_utilization metrics to debugfs References: <1546999e21a418c2510b3ed02b2b1f76b2b0f5b7.1665614216.git.alexlzhu@fb.com> Date: Tue, 18 Oct 2022 11:21:20 +0800 In-Reply-To: <1546999e21a418c2510b3ed02b2b1f76b2b0f5b7.1665614216.git.alexlzhu@fb.com> (alexlzhu@fb.com's message of "Wed, 12 Oct 2022 15:51:45 -0700") Message-ID: <87r0z5stov.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666063348; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=THyDTdKCFK7judTdCK4jzdVzf9+614gFI4Z0liWZpBI=; b=Qj3Zt5INUbU6edJB10yEkvL5ROCfYDNzAyFRZE3fp6+AzRWmFMWsK2/a7C1wQa08DwgPUc RMheB5CtG9bgLJLipO3qYqRL8fFh0BfJPhEJGTA1F/JIS8MFqL83Pt5KfhsXL2LnDQLvL1 K1/MwMnqo/ZYTXCz+x+8U5+9QFta4bI= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=ZcipxD0m; spf=pass (imf25.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666063348; a=rsa-sha256; cv=none; b=PSgxqoRuR0bgfRYKUYd/ZRCAosPE7Xac1+B9w1mxpc/RkxX15aaNXzzJZVLA/GCqfnkyln jXnFTjHJr891n9R2MnojAFRpAaTP9/Nme9MGI+76Ebceev7ZeZx8BKk7Ry0cfmXoJ/Z3na Q3AU4J5dtwdpmpbqwWkU4h7rCWjAwVE= X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 13BF4A0027 X-Rspam-User: Authentication-Results: imf25.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=ZcipxD0m; spf=pass (imf25.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Stat-Signature: rjjb4sjawwjtcedbr7axyerfjopiudkt X-HE-Tag: 1666063346-796361 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: writes: > From: Alexander Zhu > > This change introduces a tool that scans through all of physical > memory for anonymous THPs and groups them into buckets based > on utilization. It also includes an interface under > /sys/kernel/debug/thp_utilization. > > Sample Output: > > Utilized[0-50]: 1331 680884 > Utilized[51-101]: 9 3983 > Utilized[102-152]: 3 1187 > Utilized[153-203]: 0 0 > Utilized[204-255]: 2 539 > Utilized[256-306]: 5 1135 > Utilized[307-357]: 1 192 > Utilized[358-408]: 0 0 > Utilized[409-459]: 1 57 > Utilized[460-512]: 400 13 > Last Scan Time: 223.98s > Last Scan Duration: 70.65s > > This indicates that there are 1331 THPs that have between 0 and 50 > utilized (non zero) pages. In total there are 680884 zero pages in > this utilization bucket. THPs in the [0-50] bucket compose 76% of total > THPs, and are responsible for 99% of total zero pages across all > THPs. In other words, the least utilized THPs are responsible for almost > all of the memory waste when THP is always enabled. Similar results > have been observed across production workloads. > > The last two lines indicate the timestamp and duration of the most recent > scan through all of physical memory. Here we see that the last scan > occurred 223.98 seconds after boot time and took 70.65 seconds. > > Utilization of a THP is defined as the percentage of nonzero > pages in the THP. The worker thread will scan through all > of physical memory and obtain utilization of all anonymous > THPs. It will gather this information by periodically scanning > through all of physical memory for anonymous THPs, group them > into buckets based on utilization, and report utilization > information through debugfs under /sys/kernel/debug/thp_utilization. > > Signed-off-by: Alexander Zhu > --- > v1 to v2 > -reversed ordering of is_transparent_hugepage and PageAnon in is_anon_transparent_hugepage, page->mapping is only meaningful for user pages > > RFC to v1 > -Refactored out the code to obtain the thp_utilization_bucket, as that now has to be used in multiple places. > > Documentation/admin-guide/mm/transhuge.rst | 9 + > include/linux/huge_mm.h | 3 + > mm/huge_memory.c | 202 +++++++++++++++++++++ > 3 files changed, 214 insertions(+) > > diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst > index 8ee78ec232eb..21d86303c97e 100644 > --- a/Documentation/admin-guide/mm/transhuge.rst > +++ b/Documentation/admin-guide/mm/transhuge.rst > @@ -304,6 +304,15 @@ To identify what applications are mapping file transparent huge pages, it > is necessary to read ``/proc/PID/smaps`` and count the FileHugeMapped fields > for each mapping. > > +The utilization of transparent hugepages can be viewed by reading > +``/sys/kernel/debug/thp_utilization``. The utilization of a THP is defined > +as the ratio of non zero filled 4kb pages to the total number of pages in a > +THP. The buckets are labelled by the range of total utilized 4kb pages with > +one line per utilization bucket. Each line contains the total number of > +THPs in that bucket and the total number of zero filled 4kb pages summed > +over all THPs in that bucket. The last two lines show the timestamp and > +duration respectively of the most recent scan over all of physical memory. > + > Note that reading the smaps file is expensive and reading it > frequently will incur overhead. > > diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > index a1341fdcf666..13ac7b2f29ae 100644 > --- a/include/linux/huge_mm.h > +++ b/include/linux/huge_mm.h > @@ -178,6 +178,9 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, > unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, > unsigned long len, unsigned long pgoff, unsigned long flags); > > +int thp_number_utilized_pages(struct page *page); > +int thp_utilization_bucket(int num_utilized_pages); > + > void prep_transhuge_page(struct page *page); > void free_transhuge_page(struct page *page); > > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > index 1cc4a5f4791e..29e97df37c29 100644 > --- a/mm/huge_memory.c > +++ b/mm/huge_memory.c > @@ -46,6 +46,16 @@ > #define CREATE_TRACE_POINTS > #include > > +/* > + * The number of utilization buckets THPs will be grouped in > + * under /sys/kernel/debug/thp_utilization. > + */ > +#define THP_UTIL_BUCKET_NR 10 > +/* > + * The number of PFNs (and hence hugepages) to scan through on each periodic > + * run of the scanner that generates /sys/kernel/debug/thp_utilization. > + */ > +#define THP_UTIL_SCAN_SIZE 256 > /* > * By default, transparent hugepage support is disabled in order to avoid > * risking an increased memory footprint for applications that are not > @@ -71,6 +81,25 @@ static atomic_t huge_zero_refcount; > struct page *huge_zero_page __read_mostly; > unsigned long huge_zero_pfn __read_mostly = ~0UL; > > +static void thp_utilization_workfn(struct work_struct *work); > +static DECLARE_DELAYED_WORK(thp_utilization_work, thp_utilization_workfn); > + > +struct thp_scan_info_bucket { > + int nr_thps; > + int nr_zero_pages; > +}; > + > +struct thp_scan_info { > + struct thp_scan_info_bucket buckets[THP_UTIL_BUCKET_NR]; > + struct zone *scan_zone; > + struct timespec64 last_scan_duration; > + struct timespec64 last_scan_time; > + unsigned long pfn; > +}; > + > +static struct thp_scan_info thp_scan_debugfs; > +static struct thp_scan_info thp_scan; > + > bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, > bool smaps, bool in_pf, bool enforce_sysfs) > { > @@ -485,6 +514,7 @@ static int __init hugepage_init(void) > if (err) > goto err_slab; > > + schedule_delayed_work(&thp_utilization_work, HZ); > err = register_shrinker(&huge_zero_page_shrinker, "thp-zero"); > if (err) > goto err_hzp_shrinker; > @@ -599,6 +629,11 @@ static inline bool is_transparent_hugepage(struct page *page) > page[1].compound_dtor == TRANSHUGE_PAGE_DTOR; > } > > +static inline bool is_anon_transparent_hugepage(struct page *page) > +{ > + return is_transparent_hugepage(page) && PageAnon(page); > +} > + > static unsigned long __thp_get_unmapped_area(struct file *filp, > unsigned long addr, unsigned long len, > loff_t off, unsigned long flags, unsigned long size) > @@ -649,6 +684,49 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, > } > EXPORT_SYMBOL_GPL(thp_get_unmapped_area); > > +int thp_number_utilized_pages(struct page *page) > +{ > + struct folio *folio; > + unsigned long page_offset, value; > + int thp_nr_utilized_pages = HPAGE_PMD_NR; > + int step_size = sizeof(unsigned long); > + bool is_all_zeroes; > + void *kaddr; > + int i; > + > + if (!page || !is_anon_transparent_hugepage(page)) > + return -1; > + > + folio = page_folio(page); > + for (i = 0; i < folio_nr_pages(folio); i++) { > + kaddr = kmap_local_folio(folio, i); > + is_all_zeroes = true; > + for (page_offset = 0; page_offset < PAGE_SIZE; page_offset += step_size) { > + value = *(unsigned long *)(kaddr + page_offset); > + if (value != 0) { > + is_all_zeroes = false; > + break; > + } > + } > + if (is_all_zeroes) > + thp_nr_utilized_pages--; > + > + kunmap_local(kaddr); > + } > + return thp_nr_utilized_pages; > +} > + > +int thp_utilization_bucket(int num_utilized_pages) > +{ > + int bucket; > + > + if (num_utilized_pages < 0 || num_utilized_pages > HPAGE_PMD_NR) > + return -1; > + /* Group THPs into utilization buckets */ > + bucket = num_utilized_pages * THP_UTIL_BUCKET_NR / HPAGE_PMD_NR; > + return min(bucket, THP_UTIL_BUCKET_NR - 1); > +} > + > static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, > struct page *page, gfp_t gfp) > { > @@ -3174,6 +3252,42 @@ static int __init split_huge_pages_debugfs(void) > return 0; > } > late_initcall(split_huge_pages_debugfs); > + > +static int thp_utilization_show(struct seq_file *seqf, void *pos) > +{ > + int i; > + int start; > + int end; > + > + for (i = 0; i < THP_UTIL_BUCKET_NR; i++) { > + start = i * HPAGE_PMD_NR / THP_UTIL_BUCKET_NR; > + end = (i + 1 == THP_UTIL_BUCKET_NR) > + ? HPAGE_PMD_NR > + : ((i + 1) * HPAGE_PMD_NR / THP_UTIL_BUCKET_NR - 1); > + /* The last bucket will need to contain 100 */ > + seq_printf(seqf, "Utilized[%d-%d]: %d %d\n", start, end, > + thp_scan_debugfs.buckets[i].nr_thps, > + thp_scan_debugfs.buckets[i].nr_zero_pages); > + } > + seq_printf(seqf, "Last Scan Time: %lu.%02lus\n", > + (unsigned long)thp_scan_debugfs.last_scan_time.tv_sec, > + (thp_scan_debugfs.last_scan_time.tv_nsec / (NSEC_PER_SEC / 100))); > + > + seq_printf(seqf, "Last Scan Duration: %lu.%02lus\n", > + (unsigned long)thp_scan_debugfs.last_scan_duration.tv_sec, > + (thp_scan_debugfs.last_scan_duration.tv_nsec / (NSEC_PER_SEC / 100))); > + > + return 0; > +} > +DEFINE_SHOW_ATTRIBUTE(thp_utilization); > + > +static int __init thp_utilization_debugfs(void) > +{ > + debugfs_create_file("thp_utilization", 0200, NULL, NULL, > + &thp_utilization_fops); > + return 0; > +} > +late_initcall(thp_utilization_debugfs); > #endif > > #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION > @@ -3269,3 +3383,91 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) > trace_remove_migration_pmd(address, pmd_val(pmde)); > } > #endif > + > +static void thp_scan_next_zone(void) > +{ > + struct timespec64 current_time; > + int i; > + bool update_debugfs; > + /* > + * THP utilization worker thread has reached the end > + * of the memory zone. Proceed to the next zone. > + */ > + thp_scan.scan_zone = next_zone(thp_scan.scan_zone); > + update_debugfs = !thp_scan.scan_zone; > + thp_scan.scan_zone = update_debugfs ? (first_online_pgdat())->node_zones > + : thp_scan.scan_zone; > + thp_scan.pfn = (thp_scan.scan_zone->zone_start_pfn + HPAGE_PMD_NR - 1) > + & ~(HPAGE_PMD_SIZE - 1); > + if (!update_debugfs) > + return; > + /* > + * If the worker has scanned through all of physical > + * memory. Then update information displayed in /sys/kernel/debug/thp_utilization > + */ > + ktime_get_ts64(¤t_time); > + thp_scan_debugfs.last_scan_duration = timespec64_sub(current_time, > + thp_scan_debugfs.last_scan_time); > + thp_scan_debugfs.last_scan_time = current_time; > + > + for (i = 0; i < THP_UTIL_BUCKET_NR; i++) { > + thp_scan_debugfs.buckets[i].nr_thps = thp_scan.buckets[i].nr_thps; > + thp_scan_debugfs.buckets[i].nr_zero_pages = thp_scan.buckets[i].nr_zero_pages; > + thp_scan.buckets[i].nr_thps = 0; > + thp_scan.buckets[i].nr_zero_pages = 0; > + } memcpy(&thp_scan_debugfs.buckets, &thp_scan.buckets, sizeof(thp_scan.buckets)); memset(&thp_scan.buckets, 0, sizeof(thp_scan.buckets)); ? > +} > + > +static void thp_util_scan(unsigned long pfn_end) > +{ > + struct page *page = NULL; > + int bucket, num_utilized_pages, current_pfn; > + int i; > + /* > + * Scan through each memory zone in chunks of THP_UTIL_SCAN_SIZE > + * PFNs every second looking for anonymous THPs. > + */ > + for (i = 0; i < THP_UTIL_SCAN_SIZE; i++) { > + current_pfn = thp_scan.pfn; > + thp_scan.pfn += HPAGE_PMD_NR; > + if (current_pfn >= pfn_end) > + return; > + > + if (!pfn_valid(current_pfn)) > + continue; > + > + page = pfn_to_page(current_pfn); In addition to use pfn_to_online_page() as pointed out by Kirill, do we need to consider memory hot-remove too? Per my understanding, the page may be offlined under us while we are scanning, so we need to use get/put_online_mems() to enclose the code to access the page or increase the page count. But I don't find this is considered in some existing code such as split_huge_pages_all(). > + num_utilized_pages = thp_number_utilized_pages(page); > + bucket = thp_utilization_bucket(num_utilized_pages); > + if (bucket < 0) > + continue; > + > + thp_scan.buckets[bucket].nr_thps++; > + thp_scan.buckets[bucket].nr_zero_pages += (HPAGE_PMD_NR - num_utilized_pages); > + } > +} > + > +static void thp_utilization_workfn(struct work_struct *work) > +{ > + unsigned long pfn_end; > + > + if (!thp_scan.scan_zone) > + thp_scan.scan_zone = (first_online_pgdat())->node_zones; > + /* > + * Worker function that scans through all of physical memory > + * for anonymous THPs. > + */ > + pfn_end = (thp_scan.scan_zone->zone_start_pfn + > + thp_scan.scan_zone->spanned_pages + HPAGE_PMD_NR - 1) > + & ~(HPAGE_PMD_SIZE - 1); Just use pfn_end = round_up(zone_end_pfn(thp_scan.scan_zone), HPAGE_PMD_NR); But why round_up()? Just use zone_end_pfn() is sufficient? > + /* If we have reached the end of the zone or end of physical memory > + * move on to the next zone. Otherwise, scan the next PFNs in the > + * current zone. > + */ > + if (!populated_zone(thp_scan.scan_zone) || thp_scan.pfn >= pfn_end) s/populated_zone/managed_zone/ ? > + thp_scan_next_zone(); > + else > + thp_util_scan(pfn_end); > + > + schedule_delayed_work(&thp_utilization_work, HZ); > +} Best Regards, Huang, Ying