From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8356C4345F for ; Fri, 12 Apr 2024 10:25:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 20D8A6B008A; Fri, 12 Apr 2024 06:25:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BD4D6B008C; Fri, 12 Apr 2024 06:25:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 085286B0092; Fri, 12 Apr 2024 06:25:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DF4E16B008A for ; Fri, 12 Apr 2024 06:25:35 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9BF7BC0DFB for ; Fri, 12 Apr 2024 10:25:35 +0000 (UTC) X-FDA: 82000498230.10.1FF6C03 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id CD3BF1C0002 for ; Fri, 12 Apr 2024 10:25:33 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712917534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=apOEYbA38biI5zK131qyTNxKhnQALpmBmcBMbCohNBk=; b=EtH/3Pd4bWAztvFyLX0XRq0DRXF1NwOyVy3uTZdSMvvVSqWrXwrrok1ei9zXXOIEzn6jXq aI5AnHVDysoviJFCG1qaviNYyA+Ivo0WLXqDHm7QsnJqGkJ6mvSBCpMxxWZvZLGGB1lLzO P1N8L5qt2y5eTelBdTeW/7y1j/81ZXw= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712917534; a=rsa-sha256; cv=none; b=lQ/DUzhOYZS9TqPjHi5R17fzUcL6l+4zmf9nb1Uv3RdUK1LxjCwZzgDkjEMADelj5GYTdN kRJ/IgooIU077dJI7/8c4oxsdoeB+hL0cZQAHfALyxKKgtT3Zq1TSSIl5hgECEWI3dFApT Ajmw/0s28wplIVEhhXuW8Mdxxj+RqCY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 607EB339; Fri, 12 Apr 2024 03:26:02 -0700 (PDT) Received: from [10.57.73.208] (unknown [10.57.73.208]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 25C9E3F64C; Fri, 12 Apr 2024 03:25:31 -0700 (PDT) Message-ID: <744fc49e-d91b-4f5a-9a27-1a25c50c2154@arm.com> Date: Fri, 12 Apr 2024 11:25:29 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v5 1/4] mm: add per-order mTHP anon_fault_alloc and anon_fault_fallback counters Content-Language: en-GB To: Barry Song <21cnbao@gmail.com> Cc: akpm@linux-foundation.org, cerasuolodomenico@gmail.com, chrisl@kernel.org, corbet@lwn.net, david@redhat.com, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, peterx@redhat.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, yosryahmed@google.com, yuzhao@google.com References: <20240412101756.296971-1-21cnbao@gmail.com> From: Ryan Roberts In-Reply-To: <20240412101756.296971-1-21cnbao@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: CD3BF1C0002 X-Rspam-User: X-Stat-Signature: 6gce5iz15ajy8txe55icxfacgfgrtr99 X-Rspamd-Server: rspam01 X-HE-Tag: 1712917533-135354 X-HE-Meta: U2FsdGVkX18beQqZZG7zhryiRHMcJevUZWXFpLoduKHJmVaLixibf+BjOFFqbGiy87wUqLgIvvRCsj5x987ock25iLZ/hyX8tYxlbIu51epiM/6ATMgfm+zz3Laen1LakBz8Fxmx4YgYwSJJDZD2hfS/ilanOcS1z6NH5uqr2GahkOCDfagHqmWsR5dJzYOh58Sy3OeU2Rev9NbaT128vG2vgAK+F3oYoYSs5VD+TLKuUvLIsWVqW0S+MYKmz4JSfGpknA/YYxx5wFIfyMz+SqRTZVWvdDTeTSwCIDHQeN2UAnXsJbfsTZzxium7SSaci/Y1YmAjeubWsW5DGmTOGa5DONKGhSEMpZM03mR9wsGMuquumNzwqVjXZwVYzxBzr/ezkwoZQiyQqi+XK7TzHYvFbsok9BUAcOomy+zhEzHN4dxHOEy+ttj7PHueUn6O7cIx20x4Xf3oN6qGmIGq028G/Tf6rfC9UbZF/7BMDln/fARID7LcCmX83RSfh5LV25ysJoEv0tEpZtiq1CW7gc70l5OaOWF+t81aVSIHgjwmNvSOrrZ3pbsxRhh4/p5ia49ATyo6IAvlKGBrSYSqSi47m+mKY8Zl6rc08prp+K13BNM6E+K4/aglNjGDZ8JJs5Lh0pKPxNKaoDpV6H2iUlzHdyS4J9dMZDUcr8177ebPS0yuroBeVVgfpELJIhe1hjk51vMp4zK7N4CCu5BVT4j6shl/MCW+FyWqvZpMgx4x0jKHSzPvkuCRZwVAP2p8dpEue7RxWCItzXZ7MJ4e3jAkrCn3I2YDeDYT2trCWoUVAQdwohSgt3pcoLt54wolrjgcEIeOdPnV0LxNlRNraNrHJ7+hXP4nTA4CRo3s3ejEhA1DRdmGtarCSF2+vSEluQknDw+1KT2nW8TPfUbmXmQ8vPA1zCaS0ijy6Xzc9N8sbtastjxAL5jTvTi9k8KJ8GenMQJW8p/4JWoHlRk ZifYZV9v Aoif5FKkxfiKe9dnmFq6YMJWW80+YaMcbYmg6Cq00DD6uMPcgUu/sN1c0h5uZfDOfscTYiBhqfLReydCR5MsHwPnHisDaDMKTi42qQFYFO4VUwVhv+EH1XFr1kYMgEAajf3OjBvgLdmSIaZzPMWyFHvAWkRTM7wfeECje/W+wt/Ec+9FKIXr+pGtcX8bmyf+zVBjf/gEIZK6MpfHjxo/LhqLFrPoYR+wFZPGiIAqCnemEMZd3ncUlPQ3b3fIroLdjBL4VwT/BarUFpgB169E9LdtfvQzLBbquQv6ZaUM5h5pSbkuXNfl3uIiG7UqgGkqhuhIOS+PNlYQa0/48XOvvn5FmzLMyDE9MIu77g7wGSxsfK3Q0K/MW7uAgKafO+7/TvQbB6Pk67o6Bxs4JA/oqg8Lw0yIoeUFgJTOaWFicOV73EONH+BP5+3BmNWjFrL0oRInHxuYnqTDaoqcAjJOTJRjmKM3C3MlVZEce5E1opFiYKUEcYFVRS7ko6xa+R4Mp+iYL1hLRiN5ZFQDS/SkjPxL3xMavg1/eljxvLmK2nwKk30AipOwuc77yETZXNg1UMCbkGB4dm2ZgFPE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 12/04/2024 11:17, Barry Song wrote: > On Fri, Apr 12, 2024 at 9:56 PM Ryan Roberts wrote: >> >> On 12/04/2024 10:43, Barry Song wrote: >>> On Fri, Apr 12, 2024 at 9:27 PM Ryan Roberts wrote: >>>> >>>> Hi Barry, >>>> >>>> 2 remaining comments - otherwise looks good. (same comments I just made in the >>>> v4 conversation). >>>> >>>> On 12/04/2024 08:37, Barry Song wrote: >>>>> From: Barry Song >>>>> >>>>> Profiling a system blindly with mTHP has become challenging due to the >>>>> lack of visibility into its operations.  Presenting the success rate of >>>>> mTHP allocations appears to be pressing need. >>>>> >>>>> Recently, I've been experiencing significant difficulty debugging >>>>> performance improvements and regressions without these figures.  It's >>>>> crucial for us to understand the true effectiveness of mTHP in real-world >>>>> scenarios, especially in systems with fragmented memory. >>>>> >>>>> This patch establishes the framework for per-order mTHP >>>>> counters. It begins by introducing the anon_fault_alloc and >>>>> anon_fault_fallback counters. Additionally, to maintain consistency >>>>> with thp_fault_fallback_charge in /proc/vmstat, this patch also tracks >>>>> anon_fault_fallback_charge when mem_cgroup_charge fails for mTHP. >>>>> Incorporating additional counters should now be straightforward as well. >>>>> >>>>> Signed-off-by: Barry Song >>>>> Cc: Chris Li >>>>> Cc: David Hildenbrand >>>>> Cc: Domenico Cerasuolo >>>>> Cc: Kairui Song >>>>> Cc: Matthew Wilcox (Oracle) >>>>> Cc: Peter Xu >>>>> Cc: Ryan Roberts >>>>> Cc: Suren Baghdasaryan >>>>> Cc: Yosry Ahmed >>>>> Cc: Yu Zhao >>>>> --- >>>>>  include/linux/huge_mm.h | 51 ++++++++++++++++++++++++++++++++++ >>>>>  mm/huge_memory.c        | 61 +++++++++++++++++++++++++++++++++++++++++ >>>>>  mm/memory.c             |  3 ++ >>>>>  mm/page_alloc.c         |  4 +++ >>>>>  4 files changed, 119 insertions(+) >>>>> >>>>> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h >>>>> index e896ca4760f6..c5beb54b97cb 100644 >>>>> --- a/include/linux/huge_mm.h >>>>> +++ b/include/linux/huge_mm.h >>>>> @@ -264,6 +264,57 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, >>>>>                                         enforce_sysfs, orders); >>>>>  } >>>>> >>>>> +enum mthp_stat_item { >>>>> +     MTHP_STAT_ANON_FAULT_ALLOC, >>>>> +     MTHP_STAT_ANON_FAULT_FALLBACK, >>>>> +     MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, >>>>> +     __MTHP_STAT_COUNT >>>>> +}; >>>>> + >>>>> +struct mthp_stat { >>>>> +     unsigned long stats[0][__MTHP_STAT_COUNT]; >>>>> +}; >>>>> + >>>>> +extern struct mthp_stat __percpu *mthp_stats; >>>>> + >>>>> +static inline void count_mthp_stat(int order, enum mthp_stat_item item) >>>>> +{ >>>>> +     if (order <= 0 || order > PMD_ORDER || !mthp_stats) >>>>> +             return; >>>>> + >>>>> +     this_cpu_inc(mthp_stats->stats[order][item]); >>>>> +} >>>>> + >>>>> +static inline void count_mthp_stats(int order, enum mthp_stat_item item, long delta) >>>>> +{ >>>>> +     if (order <= 0 || order > PMD_ORDER || !mthp_stats) >>>>> +             return; >>>>> + >>>>> +     this_cpu_add(mthp_stats->stats[order][item], delta); >>>>> +} >>>>> + >>>>> +/* >>>>> + * Fold the foreign cpu mthp stats into our own. >>>>> + * >>>>> + * This is adding to the stats on one processor >>>>> + * but keeps the global counts constant. >>>>> + */ >>>>> +static inline void mthp_stats_fold_cpu(int cpu) >>>>> +{ >>>>> +     struct mthp_stat *fold_stat; >>>>> +     int i, j; >>>>> + >>>>> +     if (!mthp_stats) >>>>> +             return; >>>>> +     fold_stat = per_cpu_ptr(mthp_stats, cpu); >>>>> +     for (i = 1; i <= PMD_ORDER; i++) { >>>>> +             for (j = 0; j < __MTHP_STAT_COUNT; j++) { >>>>> +                     count_mthp_stats(i, j, fold_stat->stats[i][j]); >>>>> +                     fold_stat->stats[i][j] = 0; >>>>> +             } >>>>> +     } >>>>> +} >>>> >>>> This is a pretty horrible hack; I'm pretty sure just summing for all *possible* >>>> cpus should work. >>>> >>>>> + >>>>>  #define transparent_hugepage_use_zero_page()                         \ >>>>>       (transparent_hugepage_flags &                                   \ >>>>>        (1<>>>> diff --git a/mm/huge_memory.c b/mm/huge_memory.c >>>>> index dc30139590e6..21c4ac74b484 100644 >>>>> --- a/mm/huge_memory.c >>>>> +++ b/mm/huge_memory.c >>>>> @@ -526,6 +526,50 @@ static const struct kobj_type thpsize_ktype = { >>>>>       .sysfs_ops = &kobj_sysfs_ops, >>>>>  }; >>>>> >>>>> +struct mthp_stat __percpu *mthp_stats; >>>>> + >>>>> +static unsigned long sum_mthp_stat(int order, enum mthp_stat_item item) >>>>> +{ >>>>> +     unsigned long sum = 0; >>>>> +     int cpu; >>>>> + >>>>> +     cpus_read_lock(); >>>>> +     for_each_online_cpu(cpu) { >>>>> +             struct mthp_stat *this = per_cpu_ptr(mthp_stats, cpu); >>>>> + >>>>> +             sum += this->stats[order][item]; >>>>> +     } >>>>> +     cpus_read_unlock(); >>>>> + >>>>> +     return sum; >>>>> +} >>>>> + >>>>> +#define DEFINE_MTHP_STAT_ATTR(_name, _index)                                 \ >>>>> +static ssize_t _name##_show(struct kobject *kobj,                    \ >>>>> +                     struct kobj_attribute *attr, char *buf)         \ >>>>> +{                                                                    \ >>>>> +     int order = to_thpsize(kobj)->order;                            \ >>>>> +                                                                     \ >>>>> +     return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index));  \ >>>>> +}                                                                    \ >>>>> +static struct kobj_attribute _name##_attr = __ATTR_RO(_name) >>>>> + >>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC); >>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); >>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); >>>>> + >>>>> +static struct attribute *stats_attrs[] = { >>>>> +     &anon_fault_alloc_attr.attr, >>>>> +     &anon_fault_fallback_attr.attr, >>>>> +     &anon_fault_fallback_charge_attr.attr, >>>>> +     NULL, >>>>> +}; >>>>> + >>>>> +static struct attribute_group stats_attr_group = { >>>>> +     .name = "stats", >>>>> +     .attrs = stats_attrs, >>>>> +}; >>>>> + >>>>>  static struct thpsize *thpsize_create(int order, struct kobject *parent) >>>>>  { >>>>>       unsigned long size = (PAGE_SIZE << order) / SZ_1K; >>>>> @@ -549,6 +593,12 @@ static struct thpsize *thpsize_create(int order, struct kobject *parent) >>>>>               return ERR_PTR(ret); >>>>>       } >>>>> >>>>> +     ret = sysfs_create_group(&thpsize->kobj, &stats_attr_group); >>>>> +     if (ret) { >>>>> +             kobject_put(&thpsize->kobj); >>>>> +             return ERR_PTR(ret); >>>>> +     } >>>>> + >>>>>       thpsize->order = order; >>>>>       return thpsize; >>>>>  } >>>>> @@ -691,6 +741,11 @@ static int __init hugepage_init(void) >>>>>        */ >>>>>       MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER < 2); >>>>> >>>>> +     mthp_stats = __alloc_percpu((PMD_ORDER + 1) * sizeof(mthp_stats->stats[0]), >>>>> +                     sizeof(unsigned long)); >>>> >>>> Personally I think it would be cleaner to allocate statically using >>>> ilog2(MAX_PTRS_PER_PTE) instead of PMD_ORDER. >>> >>> Hi Ryan, >>> >>> I don't understand why MAX_PTRS_PER_PTE is the correct size. For ARM64, >>> >>> #define PMD_ORDER       (PMD_SHIFT - PAGE_SHIFT) >>> >>> #define MAX_PTRS_PER_PTE PTRS_PER_PTE >>> >>> #define PTRS_PER_PTE            (1 << (PAGE_SHIFT - 3)) >>> >>> while PAGE_SIZE is 16KiB or 64KiB, PTRS_PER_PTE can be a huge number? >>> >>> >>> Am I missing something? >> >> PTRS_PER_PTE is the number of PTE entries in a PTE table. On arm64 its as follows: >> >> PAGE_SIZE       PAGE_SHIFT      PTRS_PER_PTE >> 4K              12              512 >> 16K             14              2048 >> 64K             16              8192 >> >> So (PTRS_PER_PTE * PAGE_SIZE) = PMD_SIZE >> >> PMD_ORDER is ilog2(PMD_SIZE / PAGE_SIZE) = ilog2(PTRS_PER_PTE) >> >> MAX_PTRS_PER_PTE is just the maximum value that PTRS_PER_PTE will ever have, >> (and its equal to PTRS_PER_PTE except for powerpc). >> >> Pretty sure the math is correct? > > I am not convinced the math is correct :-) > > while page size is 64KiB, the page table is as below, > PMD_ORDER = L2 index bits = [41:29] = 13 != ilog2(8192) 1 << 13 = 8192 Right? So: ilog2(8192) = 13 What's wrong with that? I even checked in Python to make sure I'm not going mad: >>> import math >>> math.log2(8192) 13.0 > > > +--------+--------+--------+--------+--------+--------+--------+--------+ > |63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0| > +--------+--------+--------+--------+--------+--------+--------+--------+ > | | | | | > | | | | v > | | | | [15:0] in-page offset > | | | +----------> [28:16] L3 index > | | +--------------------------> [41:29] L2 index > | +-------------------------------> [47:42] L1 index (48-bit) > | [51:42] L1 index (52-bit) > +-------------------------------------------------> [63] TTBR0/1 > > while page size is 4KiB, the page table is as below, > > +--------+--------+--------+--------+--------+--------+--------+--------+ > |63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0| > +--------+--------+--------+--------+--------+--------+--------+--------+ > | | | | | | > | | | | | v > | | | | | [11:0] in-page offset > | | | | +-> [20:12] L3 index > | | | +-----------> [29:21] L2 index > | | +---------------------> [38:30] L1 index > | +-------------------------------> [47:39] L0 index > +-------------------------------------------------> [63] TTBR0/1 > > PMD_ORDER = L2 index bits = [29:21] = 9 = ilog2(512). > > You are only correct while page size = 4KiB. > > > >