From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 442EEC4345F for ; Fri, 12 Apr 2024 11:05:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A07E6B007B; Fri, 12 Apr 2024 07:05:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64F9F6B008A; Fri, 12 Apr 2024 07:05:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5401C6B0093; Fri, 12 Apr 2024 07:05:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2FC4B6B007B for ; Fri, 12 Apr 2024 07:05:34 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id AF0FE120E1F for ; Fri, 12 Apr 2024 11:05:33 +0000 (UTC) X-FDA: 82000598946.28.6DD3726 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf02.hostedemail.com (Postfix) with ESMTP id C9E6F8001C for ; Fri, 12 Apr 2024 11:05:30 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712919931; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7r9UT7Bm5anmJLyPtpxzK6890jC3Ihkij4wMew4H3Ak=; b=OzyN+GZ96Rb+rfHtz2f2aXQ4Ntz4RVtYaZ4qlYMXGzZ4vOrq493ZBuSoLa+SHyH1tuGkXr c5MV4loPwFYMVAO7mQztDkZ1AWi8khQ7gJ3RR66aeNxewZME7wCvuWk4jQJSbAirgvRkBl Gwa93zVYlkLIb2AHGVFRRXLfEz5SNWo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712919931; a=rsa-sha256; cv=none; b=txtDwpmjnPcwv3kfak6zJY1p2D5y2BYBEGg7EBD/wq7BikQWkWrr5N5pyAAoqpbZEVjOsS xIdowS35hqYLuuzCUaA4apCWZbl4ej7XtOGJB39SXrB8gw7vHJk2pnJWs3rQUQTZwntqL3 DIdsxfIaXY4Jp6J9/KQRY0vYsGpC35A= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3B4B6339; Fri, 12 Apr 2024 04:05:59 -0700 (PDT) Received: from [10.57.73.208] (unknown [10.57.73.208]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A8AD03F766; Fri, 12 Apr 2024 04:05:26 -0700 (PDT) Message-ID: Date: Fri, 12 Apr 2024 12:05:24 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v5 1/4] mm: add per-order mTHP anon_fault_alloc and anon_fault_fallback counters Content-Language: en-GB To: Barry Song <21cnbao@gmail.com> Cc: akpm@linux-foundation.org, cerasuolodomenico@gmail.com, chrisl@kernel.org, corbet@lwn.net, david@redhat.com, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, peterx@redhat.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, yosryahmed@google.com, yuzhao@google.com References: <20240412101756.296971-1-21cnbao@gmail.com> <744fc49e-d91b-4f5a-9a27-1a25c50c2154@arm.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: C9E6F8001C X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: k6wu3icjobma1fxpmqqs6qxqncgce9j1 X-HE-Tag: 1712919930-202565 X-HE-Meta: U2FsdGVkX1+QWtA8FZCx0e8Qsw6V/6FSu27vYl3XWdlXYVBhBg/lzbuakqvk727lEcRqWXmkAizdo8p2JqizfOPctpMCD6ZpUQadQ6L0sIBIcg/6AYw5rk4w74tVn8rQWmAg923rwGvE5QLQlfdkAyfqmXBpvKzq0i+X8/F2hsqrIxChBtEmK+ko55/rKL/c3wxOEKDO4vNNbXSXeEXNc9RpqJ7WJbh5/z1Tfkyw4WTksoAm55VGjvNLoMfUuPwGlG/vV4TTz9D+zBQR5h3gsO/6Vuog03SRApO10YoF1iif+X13ZMlR4VycwVkK5jOP8irrApZiEQ78WTZrzwNh6wX6GvhfvDYX4SZBg2bJ/f6Fn7Wfb5AYZTCQsoKlKvUCixL6E+q0F4qwIT4T3BrBAQSYSjN2WRUBbxk9OUAfMBQTcuUbbfh5DfdOApl30nrcNUDPcNiiWWbkdZwa5cMrbmxGb2wsldz0il7hoBIwTHVXq4rRbWDMKidhpLrthgzUQDyjmiKakGOb6q7g4Evm7Go9PkOWcgliwsfBnj0plYzcQenye4Q91JYXsgbJXT8oR8+DzIQSDmdnSsuJFZw0y7595yTptlwk9C7hEAG5vYX7a6V8ytK+44W/UXfsWCwtFuTklFCvVPws+VYpCEiiPeaHNW8Ymr3VP5Z71ARjuybmHXXlZdibSoFARo7nHOSUiS8xydVUMyMv+Ldm5TeMvq+Xa3NkMTRXFvC7CGBdd/McaTfR0rN57ayi6AbXZQLs3sCqXVSOPeMGJOLUsCqCNWoMzK+K5aKiFYJp+RR16CKcsYpYigN2B+kZo3OiN79jfMbqzX+vGid6cXMDwmlOwaLS+1xH/IbPRZyqkr6h9e80L5rpsNYKP3Bb9Xm9cBIR64T6huw1ZQp7vbVt0ILxLcN5xxJjuK/14sYRuSW9lT9jCFzotSQkBdX0bcaveQRrOzmncgM5As8s+Zys6tD 1B1BqY3v is63OzcpIzNzi3q165eQS6qxRMuy6Osywyd1TuZy6Y3SMCg5JE8lBbuBckOHk/N0XL6SQSX8XyOAxaZnGF9N9gT4bjrL8ZO9WLX+8tvcsqTXTDqutB27Ww6s7XxjM68dqqPu3TQdTeQ/iT/q99HEw7YxOXN9eGf44MldubLc2451+6+anpkva4S1fPb4oat9CjHJV9UNsOlV2vtoQ5/vaQQaEQ4KTWwjLrrMFdS4J67dLKSt5XYK7n4IvLIOm0AV7QOEyrdcKfJxIeLvErg3f1lMy8Km8sYjyKSrMXVMC19Ly2DFwLrM8YDuyk0nQaKw3GgO81NRh3y1DRgOdl1IdT661T6E5b3up7NlA4UOu7sX5xjLwEEfIJvdb4zz9Chf7fbmryTiEuxfRd5UzTcTz/kjv9KlwCkCCNKtkmSQiOzRQe1cdPZpGZmQ8StpmB2RoNHy97nrgXqqJL31ecMbdE8o+j2b4r2oHKT14I/2QaWX+PQWWBnHcTBZmKx7fXgtoal7DB1QRZ8zpbmxwntCA/NQgT65ADEYM0y+0vEBkn5vKeNxXup+Y9lF5o0mqOWoHLMlzuj6R9efQ/RM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 12/04/2024 11:53, Barry Song wrote: > On Fri, Apr 12, 2024 at 10:38 PM Ryan Roberts wrote: >> >> On 12/04/2024 11:29, Barry Song wrote: >>> On Fri, Apr 12, 2024 at 10:25 PM Ryan Roberts wrote: >>>> >>>> On 12/04/2024 11:17, Barry Song wrote: >>>>> On Fri, Apr 12, 2024 at 9:56 PM Ryan Roberts wrote: >>>>>> >>>>>> On 12/04/2024 10:43, Barry Song wrote: >>>>>>> On Fri, Apr 12, 2024 at 9:27 PM Ryan Roberts wrote: >>>>>>>> >>>>>>>> Hi Barry, >>>>>>>> >>>>>>>> 2 remaining comments - otherwise looks good. (same comments I just made in the >>>>>>>> v4 conversation). >>>>>>>> >>>>>>>> On 12/04/2024 08:37, Barry Song wrote: >>>>>>>>> From: Barry Song >>>>>>>>> >>>>>>>>> Profiling a system blindly with mTHP has become challenging due to the >>>>>>>>> lack of visibility into its operations. Presenting the success rate of >>>>>>>>> mTHP allocations appears to be pressing need. >>>>>>>>> >>>>>>>>> Recently, I've been experiencing significant difficulty debugging >>>>>>>>> performance improvements and regressions without these figures. It's >>>>>>>>> crucial for us to understand the true effectiveness of mTHP in real-world >>>>>>>>> scenarios, especially in systems with fragmented memory. >>>>>>>>> >>>>>>>>> This patch establishes the framework for per-order mTHP >>>>>>>>> counters. It begins by introducing the anon_fault_alloc and >>>>>>>>> anon_fault_fallback counters. Additionally, to maintain consistency >>>>>>>>> with thp_fault_fallback_charge in /proc/vmstat, this patch also tracks >>>>>>>>> anon_fault_fallback_charge when mem_cgroup_charge fails for mTHP. >>>>>>>>> Incorporating additional counters should now be straightforward as well. >>>>>>>>> >>>>>>>>> Signed-off-by: Barry Song >>>>>>>>> Cc: Chris Li >>>>>>>>> Cc: David Hildenbrand >>>>>>>>> Cc: Domenico Cerasuolo >>>>>>>>> Cc: Kairui Song >>>>>>>>> Cc: Matthew Wilcox (Oracle) >>>>>>>>> Cc: Peter Xu >>>>>>>>> Cc: Ryan Roberts >>>>>>>>> Cc: Suren Baghdasaryan >>>>>>>>> Cc: Yosry Ahmed >>>>>>>>> Cc: Yu Zhao >>>>>>>>> --- >>>>>>>>> include/linux/huge_mm.h | 51 ++++++++++++++++++++++++++++++++++ >>>>>>>>> mm/huge_memory.c | 61 +++++++++++++++++++++++++++++++++++++++++ >>>>>>>>> mm/memory.c | 3 ++ >>>>>>>>> mm/page_alloc.c | 4 +++ >>>>>>>>> 4 files changed, 119 insertions(+) >>>>>>>>> >>>>>>>>> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h >>>>>>>>> index e896ca4760f6..c5beb54b97cb 100644 >>>>>>>>> --- a/include/linux/huge_mm.h >>>>>>>>> +++ b/include/linux/huge_mm.h >>>>>>>>> @@ -264,6 +264,57 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, >>>>>>>>> enforce_sysfs, orders); >>>>>>>>> } >>>>>>>>> >>>>>>>>> +enum mthp_stat_item { >>>>>>>>> + MTHP_STAT_ANON_FAULT_ALLOC, >>>>>>>>> + MTHP_STAT_ANON_FAULT_FALLBACK, >>>>>>>>> + MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, >>>>>>>>> + __MTHP_STAT_COUNT >>>>>>>>> +}; >>>>>>>>> + >>>>>>>>> +struct mthp_stat { >>>>>>>>> + unsigned long stats[0][__MTHP_STAT_COUNT]; >>>>>>>>> +}; >>>>>>>>> + >>>>>>>>> +extern struct mthp_stat __percpu *mthp_stats; >>>>>>>>> + >>>>>>>>> +static inline void count_mthp_stat(int order, enum mthp_stat_item item) >>>>>>>>> +{ >>>>>>>>> + if (order <= 0 || order > PMD_ORDER || !mthp_stats) >>>>>>>>> + return; >>>>>>>>> + >>>>>>>>> + this_cpu_inc(mthp_stats->stats[order][item]); >>>>>>>>> +} >>>>>>>>> + >>>>>>>>> +static inline void count_mthp_stats(int order, enum mthp_stat_item item, long delta) >>>>>>>>> +{ >>>>>>>>> + if (order <= 0 || order > PMD_ORDER || !mthp_stats) >>>>>>>>> + return; >>>>>>>>> + >>>>>>>>> + this_cpu_add(mthp_stats->stats[order][item], delta); >>>>>>>>> +} >>>>>>>>> + >>>>>>>>> +/* >>>>>>>>> + * Fold the foreign cpu mthp stats into our own. >>>>>>>>> + * >>>>>>>>> + * This is adding to the stats on one processor >>>>>>>>> + * but keeps the global counts constant. >>>>>>>>> + */ >>>>>>>>> +static inline void mthp_stats_fold_cpu(int cpu) >>>>>>>>> +{ >>>>>>>>> + struct mthp_stat *fold_stat; >>>>>>>>> + int i, j; >>>>>>>>> + >>>>>>>>> + if (!mthp_stats) >>>>>>>>> + return; >>>>>>>>> + fold_stat = per_cpu_ptr(mthp_stats, cpu); >>>>>>>>> + for (i = 1; i <= PMD_ORDER; i++) { >>>>>>>>> + for (j = 0; j < __MTHP_STAT_COUNT; j++) { >>>>>>>>> + count_mthp_stats(i, j, fold_stat->stats[i][j]); >>>>>>>>> + fold_stat->stats[i][j] = 0; >>>>>>>>> + } >>>>>>>>> + } >>>>>>>>> +} >>>>>>>> >>>>>>>> This is a pretty horrible hack; I'm pretty sure just summing for all *possible* >>>>>>>> cpus should work. >>>>>>>> >>>>>>>>> + >>>>>>>>> #define transparent_hugepage_use_zero_page() \ >>>>>>>>> (transparent_hugepage_flags & \ >>>>>>>>> (1<>>>>>>>> diff --git a/mm/huge_memory.c b/mm/huge_memory.c >>>>>>>>> index dc30139590e6..21c4ac74b484 100644 >>>>>>>>> --- a/mm/huge_memory.c >>>>>>>>> +++ b/mm/huge_memory.c >>>>>>>>> @@ -526,6 +526,50 @@ static const struct kobj_type thpsize_ktype = { >>>>>>>>> .sysfs_ops = &kobj_sysfs_ops, >>>>>>>>> }; >>>>>>>>> >>>>>>>>> +struct mthp_stat __percpu *mthp_stats; >>>>>>>>> + >>>>>>>>> +static unsigned long sum_mthp_stat(int order, enum mthp_stat_item item) >>>>>>>>> +{ >>>>>>>>> + unsigned long sum = 0; >>>>>>>>> + int cpu; >>>>>>>>> + >>>>>>>>> + cpus_read_lock(); >>>>>>>>> + for_each_online_cpu(cpu) { >>>>>>>>> + struct mthp_stat *this = per_cpu_ptr(mthp_stats, cpu); >>>>>>>>> + >>>>>>>>> + sum += this->stats[order][item]; >>>>>>>>> + } >>>>>>>>> + cpus_read_unlock(); >>>>>>>>> + >>>>>>>>> + return sum; >>>>>>>>> +} >>>>>>>>> + >>>>>>>>> +#define DEFINE_MTHP_STAT_ATTR(_name, _index) \ >>>>>>>>> +static ssize_t _name##_show(struct kobject *kobj, \ >>>>>>>>> + struct kobj_attribute *attr, char *buf) \ >>>>>>>>> +{ \ >>>>>>>>> + int order = to_thpsize(kobj)->order; \ >>>>>>>>> + \ >>>>>>>>> + return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index)); \ >>>>>>>>> +} \ >>>>>>>>> +static struct kobj_attribute _name##_attr = __ATTR_RO(_name) >>>>>>>>> + >>>>>>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC); >>>>>>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); >>>>>>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); >>>>>>>>> + >>>>>>>>> +static struct attribute *stats_attrs[] = { >>>>>>>>> + &anon_fault_alloc_attr.attr, >>>>>>>>> + &anon_fault_fallback_attr.attr, >>>>>>>>> + &anon_fault_fallback_charge_attr.attr, >>>>>>>>> + NULL, >>>>>>>>> +}; >>>>>>>>> + >>>>>>>>> +static struct attribute_group stats_attr_group = { >>>>>>>>> + .name = "stats", >>>>>>>>> + .attrs = stats_attrs, >>>>>>>>> +}; >>>>>>>>> + >>>>>>>>> static struct thpsize *thpsize_create(int order, struct kobject *parent) >>>>>>>>> { >>>>>>>>> unsigned long size = (PAGE_SIZE << order) / SZ_1K; >>>>>>>>> @@ -549,6 +593,12 @@ static struct thpsize *thpsize_create(int order, struct kobject *parent) >>>>>>>>> return ERR_PTR(ret); >>>>>>>>> } >>>>>>>>> >>>>>>>>> + ret = sysfs_create_group(&thpsize->kobj, &stats_attr_group); >>>>>>>>> + if (ret) { >>>>>>>>> + kobject_put(&thpsize->kobj); >>>>>>>>> + return ERR_PTR(ret); >>>>>>>>> + } >>>>>>>>> + >>>>>>>>> thpsize->order = order; >>>>>>>>> return thpsize; >>>>>>>>> } >>>>>>>>> @@ -691,6 +741,11 @@ static int __init hugepage_init(void) >>>>>>>>> */ >>>>>>>>> MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER < 2); >>>>>>>>> >>>>>>>>> + mthp_stats = __alloc_percpu((PMD_ORDER + 1) * sizeof(mthp_stats->stats[0]), >>>>>>>>> + sizeof(unsigned long)); >>>>>>>> >>>>>>>> Personally I think it would be cleaner to allocate statically using >>>>>>>> ilog2(MAX_PTRS_PER_PTE) instead of PMD_ORDER. >>>>>>> >>>>>>> Hi Ryan, >>>>>>> >>>>>>> I don't understand why MAX_PTRS_PER_PTE is the correct size. For ARM64, >>>>>>> >>>>>>> #define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) >>>>>>> >>>>>>> #define MAX_PTRS_PER_PTE PTRS_PER_PTE >>>>>>> >>>>>>> #define PTRS_PER_PTE (1 << (PAGE_SHIFT - 3)) >>>>>>> >>>>>>> while PAGE_SIZE is 16KiB or 64KiB, PTRS_PER_PTE can be a huge number? >>>>>>> >>>>>>> >>>>>>> Am I missing something? >>>>>> >>>>>> PTRS_PER_PTE is the number of PTE entries in a PTE table. On arm64 its as follows: >>>>>> >>>>>> PAGE_SIZE PAGE_SHIFT PTRS_PER_PTE >>>>>> 4K 12 512 >>>>>> 16K 14 2048 >>>>>> 64K 16 8192 >>>>>> >>>>>> So (PTRS_PER_PTE * PAGE_SIZE) = PMD_SIZE >>>>>> >>>>>> PMD_ORDER is ilog2(PMD_SIZE / PAGE_SIZE) = ilog2(PTRS_PER_PTE) >>>>>> >>>>>> MAX_PTRS_PER_PTE is just the maximum value that PTRS_PER_PTE will ever have, >>>>>> (and its equal to PTRS_PER_PTE except for powerpc). >>>>>> >>>>>> Pretty sure the math is correct? >>>>> >>>>> I am not convinced the math is correct :-) >>>>> >>>>> while page size is 64KiB, the page table is as below, >>>>> PMD_ORDER = L2 index bits = [41:29] = 13 != ilog2(8192) >>>> >>>> 1 << 13 = 8192 >>>> >>>> Right? So: >>>> >>>> ilog2(8192) = 13 >>>> >>>> What's wrong with that? >>>> >>>> I even checked in Python to make sure I'm not going mad: >>>> >>>>>>> import math >>>>>>> math.log2(8192) >>>> 13.0 >>> >>> You're correct. My mind fixated on the '16' in the line '64K 16 8192'. >>> I mistakenly thought ilog2(8192) equals 16. Apologies for the confusion. >> >> No worries! We got there in the end :) >> >> Of course my suggestion relies on being able to get a compile-time constant from >> ilog2(MAX_PTRS_PER_PTE). I think that should work, right? > > I guess so, ilog2 can detect compile-time const, otherwise, it will find the > last (most-significant) bit set. > > I've implemented the following change, and the build all passed. > Currently conducting testing. LGTM - much cleaner! > > diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > index c5beb54b97cb..d4fdb2641070 100644 > --- a/include/linux/huge_mm.h > +++ b/include/linux/huge_mm.h > @@ -272,47 +272,17 @@ enum mthp_stat_item { > }; > > struct mthp_stat { > - unsigned long stats[0][__MTHP_STAT_COUNT]; > + unsigned long stats[ilog2(MAX_PTRS_PER_PTE) + 1][__MTHP_STAT_COUNT]; > }; > > -extern struct mthp_stat __percpu *mthp_stats; > +DECLARE_PER_CPU(struct mthp_stat, mthp_stats); > > static inline void count_mthp_stat(int order, enum mthp_stat_item item) > { > - if (order <= 0 || order > PMD_ORDER || !mthp_stats) > + if (order <= 0 || order > PMD_ORDER) > return; > > - this_cpu_inc(mthp_stats->stats[order][item]); > -} > - > -static inline void count_mthp_stats(int order, enum mthp_stat_item > item, long delta) > -{ > - if (order <= 0 || order > PMD_ORDER || !mthp_stats) > - return; > - > - this_cpu_add(mthp_stats->stats[order][item], delta); > -} > - > -/* > - * Fold the foreign cpu mthp stats into our own. > - * > - * This is adding to the stats on one processor > - * but keeps the global counts constant. > - */ > -static inline void mthp_stats_fold_cpu(int cpu) > -{ > - struct mthp_stat *fold_stat; > - int i, j; > - > - if (!mthp_stats) > - return; > - fold_stat = per_cpu_ptr(mthp_stats, cpu); > - for (i = 1; i <= PMD_ORDER; i++) { > - for (j = 0; j < __MTHP_STAT_COUNT; j++) { > - count_mthp_stats(i, j, fold_stat->stats[i][j]); > - fold_stat->stats[i][j] = 0; > - } > - } > + this_cpu_inc(mthp_stats.stats[order][item]); > } > > #define transparent_hugepage_use_zero_page() \ > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > index 21c4ac74b484..e88961ffc398 100644 > --- a/mm/huge_memory.c > +++ b/mm/huge_memory.c > @@ -526,20 +526,18 @@ static const struct kobj_type thpsize_ktype = { > .sysfs_ops = &kobj_sysfs_ops, > }; > > -struct mthp_stat __percpu *mthp_stats; > +DEFINE_PER_CPU(struct mthp_stat, mthp_stats) = {{{0}}}; > > static unsigned long sum_mthp_stat(int order, enum mthp_stat_item item) > { > unsigned long sum = 0; > int cpu; > > - cpus_read_lock(); > - for_each_online_cpu(cpu) { > - struct mthp_stat *this = per_cpu_ptr(mthp_stats, cpu); > + for_each_possible_cpu(cpu) { > + struct mthp_stat *this = &per_cpu(mthp_stats, cpu); > > sum += this->stats[order][item]; > } > - cpus_read_unlock(); > > return sum; > } > @@ -741,11 +739,6 @@ static int __init hugepage_init(void) > */ > MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER < 2); > > - mthp_stats = __alloc_percpu((PMD_ORDER + 1) * > sizeof(mthp_stats->stats[0]), > - sizeof(unsigned long)); > - if (!mthp_stats) > - return -ENOMEM; > - > err = hugepage_init_sysfs(&hugepage_kobj); > if (err) > goto err_sysfs; > @@ -780,8 +773,6 @@ static int __init hugepage_init(void) > err_slab: > hugepage_exit_sysfs(hugepage_kobj); > err_sysfs: > - free_percpu(mthp_stats); > - mthp_stats = NULL; > return err; > } > subsys_initcall(hugepage_init); > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 3135b5ca2457..b51becf03d1e 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -5840,10 +5840,6 @@ static int page_alloc_cpu_dead(unsigned int cpu) > */ > vm_events_fold_cpu(cpu); > > -#ifdef CONFIG_TRANSPARENT_HUGEPAGE > - mthp_stats_fold_cpu(cpu); > -#endif > - > /* > * Zero the differential counters of the dead processor > * so that the vm statistics are consistent. > > > Thanks > Barry