From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A2F6C4345F for ; Fri, 12 Apr 2024 10:53:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF92E6B008C; Fri, 12 Apr 2024 06:53:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EA8D46B0093; Fri, 12 Apr 2024 06:53:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D494D6B0095; Fri, 12 Apr 2024 06:53:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B277B6B008C for ; Fri, 12 Apr 2024 06:53:20 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 688001A0DD4 for ; Fri, 12 Apr 2024 10:53:20 +0000 (UTC) X-FDA: 82000568160.03.636C36D Received: from mail-vk1-f181.google.com (mail-vk1-f181.google.com [209.85.221.181]) by imf14.hostedemail.com (Postfix) with ESMTP id BC7ED10000A for ; Fri, 12 Apr 2024 10:53:18 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aOLBRtYj; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712919198; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KNMd+gb2+kz8eMgsr/jsH6tYZ45z+m5SXLIFNnje5mc=; b=YnmewBnolv95wKFTBpVhVMZKbiDBGF1JQq5FilwGsbYenHI/2wwapEx7UGNFD0+jY/9khA YSPTD85xbAOsJ/ocuCAezaWmNGIX8hxQm6DZ+KnkEmq2BLKjFjbwVo0vX/6B6wlXu+AYfD MAO6UzMwCjpppUnAcfELGj4TN6+b5PU= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aOLBRtYj; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712919198; a=rsa-sha256; cv=none; b=Vw6JoYkOwniXASoz9u4bXg9pWrhcaUjJB5Emf/kYuVkCnjqXaV/1NnIOaX6wmOjDszZjpE 2IoPr95mHlSETl9KkW/xepUNDXCk7e77QdH3LgT0lVj0zrK0fnCU16dGmmxsaU395PYwDy CkNqwEAImgBvoG1f9Uy+1CGZiNYheIc= Received: by mail-vk1-f181.google.com with SMTP id 71dfb90a1353d-4dad331b828so246467e0c.2 for ; Fri, 12 Apr 2024 03:53:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712919198; x=1713523998; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=KNMd+gb2+kz8eMgsr/jsH6tYZ45z+m5SXLIFNnje5mc=; b=aOLBRtYjS6mDmgqHKF1VHZNO9VupZBt7YerX1UwvR1XBex7FwoCUGuqOKGVNojmrW5 ZL3CpoOhOJTvzfhT5sk1L1iyXCR3s1dsqbkaAijkq858xjBAN1DqpGuDHbeygfh2mo37 98u077xqa7qtDO9KaF/1ozgEYwv4PmwwjR/gd2utLdN2G4KsIuDaxn/jOQoOL+OZuAX0 OxhFdgx5PmnXnGhjaO3dgN1s01fyilm8t3KTHYzWj1YiDyz37BOD4FljLb4VUpfnT+yA WJy9y1JeH3dPcNGgImJbdk1QEznzeOC/dBIl/6yhwqxDZc0XY+wA0d9aWZI+dPACGD2E LuwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712919198; x=1713523998; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KNMd+gb2+kz8eMgsr/jsH6tYZ45z+m5SXLIFNnje5mc=; b=QKLFA6ErqFe5RFFF4ezdBGTq+dfGD4i8/T1CTNNdXXup3G3dyrqZ2hPYdJ7jjfA+zQ M3VctJnX9NvJ2sQ7sTu7UmgrvrVecNzsEkQyGyxo46vlYBcJvz8DcdnLxMzg6w3HAmvR GYhFCdn87gZyiOIDJSgqEJk9x0TmCIXdV+0n/8r51KqfkvcLz84SHNMXPynPBONy8JOf 8WIN1gcKwPLEYVM2R+t8NunLAj+WYuma3pHOYikb/COEuTd9Wv0brIWS+qMDoL3wqzWx XMfjLoXbgr24AhEJW5YVUXaPaytmLs7jTGTUnEDK+TIKlTU7XeX5NCzxQq/EMPVEsJ0u TxWg== X-Forwarded-Encrypted: i=1; AJvYcCWFmTJAkQvFeXA0DcRcpEygAjUV0C0WNniUx+yNsrlk3zSKmi7NlbKZC9+XXo38j+9e/5dplv1sy9atmaMxbEBgTcQ= X-Gm-Message-State: AOJu0YxpohKxUd/t9bqgWC3UN98qdoRR2zfFKq5ag45yEE09D6o7kLZ1 J6a4CkDLh8FoUfSWEpxluwmaeNEV4HMqIRvuuNm2xU8GjCejWoPYWX64dO7S/NRAAC95e3f+5Tq 3nRG6h/hSmKptMarM+jV8pd18Wdo= X-Google-Smtp-Source: AGHT+IEaiUVcWmRtkTl6RMCRUCIX5BdO+5bUQ0QPL9n4NrV/1fcO4HPLsB8YO/nf4pqjlpnD67bdYDJ7pYU1dKGicOw= X-Received: by 2002:a05:6122:4687:b0:4d4:2fbc:e61f with SMTP id di7-20020a056122468700b004d42fbce61fmr2231946vkb.14.1712919197509; Fri, 12 Apr 2024 03:53:17 -0700 (PDT) MIME-Version: 1.0 References: <20240412101756.296971-1-21cnbao@gmail.com> <744fc49e-d91b-4f5a-9a27-1a25c50c2154@arm.com> In-Reply-To: From: Barry Song <21cnbao@gmail.com> Date: Fri, 12 Apr 2024 22:53:06 +1200 Message-ID: Subject: Re: [PATCH v5 1/4] mm: add per-order mTHP anon_fault_alloc and anon_fault_fallback counters To: Ryan Roberts Cc: akpm@linux-foundation.org, cerasuolodomenico@gmail.com, chrisl@kernel.org, corbet@lwn.net, david@redhat.com, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, peterx@redhat.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, yosryahmed@google.com, yuzhao@google.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: BC7ED10000A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: zw9p7qwmoof1fuffgshqfr1bj946fiit X-HE-Tag: 1712919198-629446 X-HE-Meta: U2FsdGVkX199wjfhF/YpsdfDJHMu5LQTLtHhXfIUxKKjEgItxNv1FIOOeDOHZogSDA/sWnmI1pZY+yTJefRShpfMx+Up0UjpeFhxGA4ouZR91fytZwi85DbY66VXk2pZm4Oiq8ihHYF6nYysnzGULVKaF7rPPqlwfnvIhmnqUhiO7LGb1XuFdpQ7tjROebrdnIr65ZLirymiyyyT8AhvIkHiFh0nIXbBove70wBiTqLV/IxdGxKnt4Xl+lQI9EMzSP+CPdZQXDiNCabHVsZsklLGKaiD+77a1PZE5brlqETDV2ZViVAwECrWvnb5wEVylH5qFrsZHYsO6aBM8F5FRiaVsr9plecmhsg0MBh/aVq4q/aEViHHy8No7CjgIn95iPdVi0H3ua/AQb7GG9WT4B9Ds+YGE+EdtFHDFE2KpWMjuJYksE1kcKk9irPpeiyUiZY223xlBJR3d7qxJZif+2cgwrBf169qDe+bfud7ajSmNgq2bVuCLNdhwQDXRVdXv996fubyeXBDxEbzVSXwD1FI+b4Ey3yNAaF6ZmhS+UJEKF8SxS/QoT2P0iJe5t+istP7o07M9wFA2uvFVYgH2+AP6BjXVJ/MNy22r+QlUVKCd3p7IV2fQR0942TFODMViN7ObJSETkSP/qD7FoW8fa6of9d02ARnOXA2diojCw+uHBbHmErgIW5A1wrOYMMoRfZ8biCYPnN4O2uKt6NOPmAK92uCN8+GC7HpQWGG4GX8hlQvMnCXWYSCjQGUFsoCGZOAIsPuEBB62wvB6dQa9rniCTyl0bbPvOiK8VdLYCuOVJnv9UxeZ7zGu2n+ORaa+npOWvjYU+R7YCGEssnPva/aOmrpOTm+pdIeoijLEzSKb755DKaWxaJEIxuWY/bfjM+w9GAhuaBHGWBqzBi4MxGYemcKOkan30GIzbTAq17LGGyxA8wVNzRXmVkOq7EQ3SNC3NyCfDaSHupFbLq 2XmhLako G5dWp6jIHfyF+bAlhBJjhmBuRUBHmcUf8YM0JLIS1r36Hcllgo0SPaAUe5lmDu1u7kZSIhwfzNyxbWyNdZfTm/rNZUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Apr 12, 2024 at 10:38=E2=80=AFPM Ryan Roberts wrote: > > On 12/04/2024 11:29, Barry Song wrote: > > On Fri, Apr 12, 2024 at 10:25=E2=80=AFPM Ryan Roberts wrote: > >> > >> On 12/04/2024 11:17, Barry Song wrote: > >>> On Fri, Apr 12, 2024 at 9:56=E2=80=AFPM Ryan Roberts wrote: > >>>> > >>>> On 12/04/2024 10:43, Barry Song wrote: > >>>>> On Fri, Apr 12, 2024 at 9:27=E2=80=AFPM Ryan Roberts wrote: > >>>>>> > >>>>>> Hi Barry, > >>>>>> > >>>>>> 2 remaining comments - otherwise looks good. (same comments I just= made in the > >>>>>> v4 conversation). > >>>>>> > >>>>>> On 12/04/2024 08:37, Barry Song wrote: > >>>>>>> From: Barry Song > >>>>>>> > >>>>>>> Profiling a system blindly with mTHP has become challenging due t= o the > >>>>>>> lack of visibility into its operations. Presenting the success r= ate of > >>>>>>> mTHP allocations appears to be pressing need. > >>>>>>> > >>>>>>> Recently, I've been experiencing significant difficulty debugging > >>>>>>> performance improvements and regressions without these figures. = It's > >>>>>>> crucial for us to understand the true effectiveness of mTHP in re= al-world > >>>>>>> scenarios, especially in systems with fragmented memory. > >>>>>>> > >>>>>>> This patch establishes the framework for per-order mTHP > >>>>>>> counters. It begins by introducing the anon_fault_alloc and > >>>>>>> anon_fault_fallback counters. Additionally, to maintain consisten= cy > >>>>>>> with thp_fault_fallback_charge in /proc/vmstat, this patch also t= racks > >>>>>>> anon_fault_fallback_charge when mem_cgroup_charge fails for mTHP. > >>>>>>> Incorporating additional counters should now be straightforward a= s well. > >>>>>>> > >>>>>>> Signed-off-by: Barry Song > >>>>>>> Cc: Chris Li > >>>>>>> Cc: David Hildenbrand > >>>>>>> Cc: Domenico Cerasuolo > >>>>>>> Cc: Kairui Song > >>>>>>> Cc: Matthew Wilcox (Oracle) > >>>>>>> Cc: Peter Xu > >>>>>>> Cc: Ryan Roberts > >>>>>>> Cc: Suren Baghdasaryan > >>>>>>> Cc: Yosry Ahmed > >>>>>>> Cc: Yu Zhao > >>>>>>> --- > >>>>>>> include/linux/huge_mm.h | 51 ++++++++++++++++++++++++++++++++++ > >>>>>>> mm/huge_memory.c | 61 +++++++++++++++++++++++++++++++++++= ++++++ > >>>>>>> mm/memory.c | 3 ++ > >>>>>>> mm/page_alloc.c | 4 +++ > >>>>>>> 4 files changed, 119 insertions(+) > >>>>>>> > >>>>>>> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > >>>>>>> index e896ca4760f6..c5beb54b97cb 100644 > >>>>>>> --- a/include/linux/huge_mm.h > >>>>>>> +++ b/include/linux/huge_mm.h > >>>>>>> @@ -264,6 +264,57 @@ unsigned long thp_vma_allowable_orders(struc= t vm_area_struct *vma, > >>>>>>> enforce_sysfs, orders); > >>>>>>> } > >>>>>>> > >>>>>>> +enum mthp_stat_item { > >>>>>>> + MTHP_STAT_ANON_FAULT_ALLOC, > >>>>>>> + MTHP_STAT_ANON_FAULT_FALLBACK, > >>>>>>> + MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, > >>>>>>> + __MTHP_STAT_COUNT > >>>>>>> +}; > >>>>>>> + > >>>>>>> +struct mthp_stat { > >>>>>>> + unsigned long stats[0][__MTHP_STAT_COUNT]; > >>>>>>> +}; > >>>>>>> + > >>>>>>> +extern struct mthp_stat __percpu *mthp_stats; > >>>>>>> + > >>>>>>> +static inline void count_mthp_stat(int order, enum mthp_stat_ite= m item) > >>>>>>> +{ > >>>>>>> + if (order <=3D 0 || order > PMD_ORDER || !mthp_stats) > >>>>>>> + return; > >>>>>>> + > >>>>>>> + this_cpu_inc(mthp_stats->stats[order][item]); > >>>>>>> +} > >>>>>>> + > >>>>>>> +static inline void count_mthp_stats(int order, enum mthp_stat_it= em item, long delta) > >>>>>>> +{ > >>>>>>> + if (order <=3D 0 || order > PMD_ORDER || !mthp_stats) > >>>>>>> + return; > >>>>>>> + > >>>>>>> + this_cpu_add(mthp_stats->stats[order][item], delta); > >>>>>>> +} > >>>>>>> + > >>>>>>> +/* > >>>>>>> + * Fold the foreign cpu mthp stats into our own. > >>>>>>> + * > >>>>>>> + * This is adding to the stats on one processor > >>>>>>> + * but keeps the global counts constant. > >>>>>>> + */ > >>>>>>> +static inline void mthp_stats_fold_cpu(int cpu) > >>>>>>> +{ > >>>>>>> + struct mthp_stat *fold_stat; > >>>>>>> + int i, j; > >>>>>>> + > >>>>>>> + if (!mthp_stats) > >>>>>>> + return; > >>>>>>> + fold_stat =3D per_cpu_ptr(mthp_stats, cpu); > >>>>>>> + for (i =3D 1; i <=3D PMD_ORDER; i++) { > >>>>>>> + for (j =3D 0; j < __MTHP_STAT_COUNT; j++) { > >>>>>>> + count_mthp_stats(i, j, fold_stat->stats[i][= j]); > >>>>>>> + fold_stat->stats[i][j] =3D 0; > >>>>>>> + } > >>>>>>> + } > >>>>>>> +} > >>>>>> > >>>>>> This is a pretty horrible hack; I'm pretty sure just summing for a= ll *possible* > >>>>>> cpus should work. > >>>>>> > >>>>>>> + > >>>>>>> #define transparent_hugepage_use_zero_page() = \ > >>>>>>> (transparent_hugepage_flags & = \ > >>>>>>> (1< >>>>>>> diff --git a/mm/huge_memory.c b/mm/huge_memory.c > >>>>>>> index dc30139590e6..21c4ac74b484 100644 > >>>>>>> --- a/mm/huge_memory.c > >>>>>>> +++ b/mm/huge_memory.c > >>>>>>> @@ -526,6 +526,50 @@ static const struct kobj_type thpsize_ktype = =3D { > >>>>>>> .sysfs_ops =3D &kobj_sysfs_ops, > >>>>>>> }; > >>>>>>> > >>>>>>> +struct mthp_stat __percpu *mthp_stats; > >>>>>>> + > >>>>>>> +static unsigned long sum_mthp_stat(int order, enum mthp_stat_ite= m item) > >>>>>>> +{ > >>>>>>> + unsigned long sum =3D 0; > >>>>>>> + int cpu; > >>>>>>> + > >>>>>>> + cpus_read_lock(); > >>>>>>> + for_each_online_cpu(cpu) { > >>>>>>> + struct mthp_stat *this =3D per_cpu_ptr(mthp_stats, = cpu); > >>>>>>> + > >>>>>>> + sum +=3D this->stats[order][item]; > >>>>>>> + } > >>>>>>> + cpus_read_unlock(); > >>>>>>> + > >>>>>>> + return sum; > >>>>>>> +} > >>>>>>> + > >>>>>>> +#define DEFINE_MTHP_STAT_ATTR(_name, _index) = \ > >>>>>>> +static ssize_t _name##_show(struct kobject *kobj, = \ > >>>>>>> + struct kobj_attribute *attr, char *buf) = \ > >>>>>>> +{ = \ > >>>>>>> + int order =3D to_thpsize(kobj)->order; = \ > >>>>>>> + = \ > >>>>>>> + return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index= )); \ > >>>>>>> +} = \ > >>>>>>> +static struct kobj_attribute _name##_attr =3D __ATTR_RO(_name) > >>>>>>> + > >>>>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALL= OC); > >>>>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_= FALLBACK); > >>>>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON= _FAULT_FALLBACK_CHARGE); > >>>>>>> + > >>>>>>> +static struct attribute *stats_attrs[] =3D { > >>>>>>> + &anon_fault_alloc_attr.attr, > >>>>>>> + &anon_fault_fallback_attr.attr, > >>>>>>> + &anon_fault_fallback_charge_attr.attr, > >>>>>>> + NULL, > >>>>>>> +}; > >>>>>>> + > >>>>>>> +static struct attribute_group stats_attr_group =3D { > >>>>>>> + .name =3D "stats", > >>>>>>> + .attrs =3D stats_attrs, > >>>>>>> +}; > >>>>>>> + > >>>>>>> static struct thpsize *thpsize_create(int order, struct kobject = *parent) > >>>>>>> { > >>>>>>> unsigned long size =3D (PAGE_SIZE << order) / SZ_1K; > >>>>>>> @@ -549,6 +593,12 @@ static struct thpsize *thpsize_create(int or= der, struct kobject *parent) > >>>>>>> return ERR_PTR(ret); > >>>>>>> } > >>>>>>> > >>>>>>> + ret =3D sysfs_create_group(&thpsize->kobj, &stats_attr_grou= p); > >>>>>>> + if (ret) { > >>>>>>> + kobject_put(&thpsize->kobj); > >>>>>>> + return ERR_PTR(ret); > >>>>>>> + } > >>>>>>> + > >>>>>>> thpsize->order =3D order; > >>>>>>> return thpsize; > >>>>>>> } > >>>>>>> @@ -691,6 +741,11 @@ static int __init hugepage_init(void) > >>>>>>> */ > >>>>>>> MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER < 2); > >>>>>>> > >>>>>>> + mthp_stats =3D __alloc_percpu((PMD_ORDER + 1) * sizeof(mthp= _stats->stats[0]), > >>>>>>> + sizeof(unsigned long)); > >>>>>> > >>>>>> Personally I think it would be cleaner to allocate statically usin= g > >>>>>> ilog2(MAX_PTRS_PER_PTE) instead of PMD_ORDER. > >>>>> > >>>>> Hi Ryan, > >>>>> > >>>>> I don't understand why MAX_PTRS_PER_PTE is the correct size. For AR= M64, > >>>>> > >>>>> #define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) > >>>>> > >>>>> #define MAX_PTRS_PER_PTE PTRS_PER_PTE > >>>>> > >>>>> #define PTRS_PER_PTE (1 << (PAGE_SHIFT - 3)) > >>>>> > >>>>> while PAGE_SIZE is 16KiB or 64KiB, PTRS_PER_PTE can be a huge numbe= r? > >>>>> > >>>>> > >>>>> Am I missing something? > >>>> > >>>> PTRS_PER_PTE is the number of PTE entries in a PTE table. On arm64 i= ts as follows: > >>>> > >>>> PAGE_SIZE PAGE_SHIFT PTRS_PER_PTE > >>>> 4K 12 512 > >>>> 16K 14 2048 > >>>> 64K 16 8192 > >>>> > >>>> So (PTRS_PER_PTE * PAGE_SIZE) =3D PMD_SIZE > >>>> > >>>> PMD_ORDER is ilog2(PMD_SIZE / PAGE_SIZE) =3D ilog2(PTRS_PER_PTE) > >>>> > >>>> MAX_PTRS_PER_PTE is just the maximum value that PTRS_PER_PTE will ev= er have, > >>>> (and its equal to PTRS_PER_PTE except for powerpc). > >>>> > >>>> Pretty sure the math is correct? > >>> > >>> I am not convinced the math is correct :-) > >>> > >>> while page size is 64KiB, the page table is as below, > >>> PMD_ORDER =3D L2 index bits =3D [41:29] =3D 13 !=3D ilog2(8192) > >> > >> 1 << 13 =3D 8192 > >> > >> Right? So: > >> > >> ilog2(8192) =3D 13 > >> > >> What's wrong with that? > >> > >> I even checked in Python to make sure I'm not going mad: > >> > >>>>> import math > >>>>> math.log2(8192) > >> 13.0 > > > > You're correct. My mind fixated on the '16' in the line '64K 16 8192'. > > I mistakenly thought ilog2(8192) equals 16. Apologies for the confusion= . > > No worries! We got there in the end :) > > Of course my suggestion relies on being able to get a compile-time consta= nt from > ilog2(MAX_PTRS_PER_PTE). I think that should work, right? I guess so, ilog2 can detect compile-time const, otherwise, it will find th= e last (most-significant) bit set. I've implemented the following change, and the build all passed. Currently conducting testing. diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c5beb54b97cb..d4fdb2641070 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -272,47 +272,17 @@ enum mthp_stat_item { }; struct mthp_stat { - unsigned long stats[0][__MTHP_STAT_COUNT]; + unsigned long stats[ilog2(MAX_PTRS_PER_PTE) + 1][__MTHP_STAT_COUNT]= ; }; -extern struct mthp_stat __percpu *mthp_stats; +DECLARE_PER_CPU(struct mthp_stat, mthp_stats); static inline void count_mthp_stat(int order, enum mthp_stat_item item) { - if (order <=3D 0 || order > PMD_ORDER || !mthp_stats) + if (order <=3D 0 || order > PMD_ORDER) return; - this_cpu_inc(mthp_stats->stats[order][item]); -} - -static inline void count_mthp_stats(int order, enum mthp_stat_item item, long delta) -{ - if (order <=3D 0 || order > PMD_ORDER || !mthp_stats) - return; - - this_cpu_add(mthp_stats->stats[order][item], delta); -} - -/* - * Fold the foreign cpu mthp stats into our own. - * - * This is adding to the stats on one processor - * but keeps the global counts constant. - */ -static inline void mthp_stats_fold_cpu(int cpu) -{ - struct mthp_stat *fold_stat; - int i, j; - - if (!mthp_stats) - return; - fold_stat =3D per_cpu_ptr(mthp_stats, cpu); - for (i =3D 1; i <=3D PMD_ORDER; i++) { - for (j =3D 0; j < __MTHP_STAT_COUNT; j++) { - count_mthp_stats(i, j, fold_stat->stats[i][j]); - fold_stat->stats[i][j] =3D 0; - } - } + this_cpu_inc(mthp_stats.stats[order][item]); } #define transparent_hugepage_use_zero_page() \ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 21c4ac74b484..e88961ffc398 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -526,20 +526,18 @@ static const struct kobj_type thpsize_ktype =3D { .sysfs_ops =3D &kobj_sysfs_ops, }; -struct mthp_stat __percpu *mthp_stats; +DEFINE_PER_CPU(struct mthp_stat, mthp_stats) =3D {{{0}}}; static unsigned long sum_mthp_stat(int order, enum mthp_stat_item item) { unsigned long sum =3D 0; int cpu; - cpus_read_lock(); - for_each_online_cpu(cpu) { - struct mthp_stat *this =3D per_cpu_ptr(mthp_stats, cpu); + for_each_possible_cpu(cpu) { + struct mthp_stat *this =3D &per_cpu(mthp_stats, cpu); sum +=3D this->stats[order][item]; } - cpus_read_unlock(); return sum; } @@ -741,11 +739,6 @@ static int __init hugepage_init(void) */ MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER < 2); - mthp_stats =3D __alloc_percpu((PMD_ORDER + 1) * sizeof(mthp_stats->stats[0]), - sizeof(unsigned long)); - if (!mthp_stats) - return -ENOMEM; - err =3D hugepage_init_sysfs(&hugepage_kobj); if (err) goto err_sysfs; @@ -780,8 +773,6 @@ static int __init hugepage_init(void) err_slab: hugepage_exit_sysfs(hugepage_kobj); err_sysfs: - free_percpu(mthp_stats); - mthp_stats =3D NULL; return err; } subsys_initcall(hugepage_init); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3135b5ca2457..b51becf03d1e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5840,10 +5840,6 @@ static int page_alloc_cpu_dead(unsigned int cpu) */ vm_events_fold_cpu(cpu); -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - mthp_stats_fold_cpu(cpu); -#endif - /* * Zero the differential counters of the dead processor * so that the vm statistics are consistent. Thanks Barry