From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95734C04FFE for ; Fri, 12 Apr 2024 10:38:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E0F16B0092; Fri, 12 Apr 2024 06:38:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 090BC6B0093; Fri, 12 Apr 2024 06:38:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E99DB6B0095; Fri, 12 Apr 2024 06:38:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CBD026B0092 for ; Fri, 12 Apr 2024 06:38:54 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 77A131C1361 for ; Fri, 12 Apr 2024 10:38:54 +0000 (UTC) X-FDA: 82000531788.13.6617D11 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf05.hostedemail.com (Postfix) with ESMTP id B3A00100011 for ; Fri, 12 Apr 2024 10:38:52 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf05.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712918332; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ABYJ3x4esohm6Rm8Fds6jCBroSBo/R1XXQtE5ZFNkhE=; b=5lqmeRmU2vhRbV3FlLbJ8rKQxGihpQ+TYa+tAn6jO2IUlDQqf2qGrCwCHuwSrKPwAhY4Ox EPQ1UwUsGmEkYrIbedyU+HnJhxZGrcgAZThe3vkR8pGZS6FlGSylLqnBY229HHvTaFhXiV V9hm4wu9trc+I9e9mgpLGcPkVc5HWlo= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf05.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712918332; a=rsa-sha256; cv=none; b=2m6Q8EzMg4jImTWqKGKOVwekkC4bvX8S6ZZ6tx/Zgb3FSx6m9fri/6wOhFPGhhFWUaJvI2 Sp1II4+x1SsxOqu+ZsqHuSuLhNpmaMXQA2M0C9jwlljHhWJokCofekc0KPIuFkaRmsSNv2 UIO2ReKF9flYSVaKOFFtni1b7WnK+4Q= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3F88F113E; Fri, 12 Apr 2024 03:39:21 -0700 (PDT) Received: from [10.57.73.208] (unknown [10.57.73.208]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0159D3F64C; Fri, 12 Apr 2024 03:38:49 -0700 (PDT) Message-ID: Date: Fri, 12 Apr 2024 11:38:48 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v5 1/4] mm: add per-order mTHP anon_fault_alloc and anon_fault_fallback counters Content-Language: en-GB To: Barry Song <21cnbao@gmail.com> Cc: akpm@linux-foundation.org, cerasuolodomenico@gmail.com, chrisl@kernel.org, corbet@lwn.net, david@redhat.com, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, peterx@redhat.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, yosryahmed@google.com, yuzhao@google.com References: <20240412101756.296971-1-21cnbao@gmail.com> <744fc49e-d91b-4f5a-9a27-1a25c50c2154@arm.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: B3A00100011 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 8e5zhyi4brxqn3f891643aa85h7oz1ek X-HE-Tag: 1712918332-923099 X-HE-Meta: U2FsdGVkX18WVG3mQwCGEybyGF1uR1icMN77r9I1UcItNuK7ghVRKQ9QtAijvRtaTD+JxWjScOkk5FCjaRm/r/z+GNmC9y1vlzVoHfLBF4hlhqgC3zBjVy/Ogwt4G2Lj0Kdk3BQCVYpJKyUxKFxYy/9qKZMF0S6pdQw9qgP93vM4En8mpY9poDOtcDyhlks4+jzHaNCrNgMGoop2fR1rYsYu69vjO1buTiWybb5rqKwElfwyznCXwU1huK2l0EqcVn+A1SlWSmywB6wisgSpdCClwo7Qc8kGk5T7unCWg7ppcUH5hvT/uKoQlr93xjfCQ55Fx16IFYjFw2N0KX6uzGYEKMjfDnHru8LHACdp2p7PHDVCEaK4PVEa+KoyWDnoHDzu38SiweHxSeca5lt/+gcITvDm0VWpNiIL2iCh8l1YNXSYsYhAZ2V8JEDLZAgxOFU0Oi7bfr25U9LFNTuGo4fNc1ZFx5Qg40uT5GvFb0QMv+eeTGZkW24ebPWvlHU3PE+G1C6OoyNB87omRuaBGkZGJZthMrdm64u3hLMx/vWzwx5z2udEyQwgdjuuPeG9Aw/HT6xYOmjMcnug6ObKWsaHXHbjLdNFZ8mSCcNzQk9zbzD9rMC9byKEIhmwt1j8Q+lJyBMtlUKP2nlS7+zD+D8ktS8VAy3BREFXkjtD3HxFD6JnBc+sw2EjlOlH0DuvF3nypVJdVZHzUB3yOewNNu+uRAg2trwae6qW3BEZnswX1OjaLOjY5yFIvKZLnE/6e/8gue5hZxrYN1fCvkr6EoYu5zgc0lTPPW1xEsN8wMIGqHCAPhrG1PQjkn8lIpbo2TlkjKxClnKDyJq6U50yaSxNBWT+prKN/A0CBroS0nfLJhKrDIFuqWZkVkAQp+vgat8FDyQDA3sQsCR4LrYGhM7kwcQRHTPoKAQUYcV/ZEk2NFJvCmmO5HxjztgTlscnQGg6Z+mF7VlP5Yi2/H3 xGe+xPKZ iTnwmR2kv+q81oTl3WsRizABWuim+3scL7+q+Ey5LcYaBC07v0H8JpWZY5z2f8MUZcAFT6Be9rTqo7GX1otwjn/FcDzfhOURc4gFtC8+gkcGEcCBpSvPCCezYHuw97RbxoCNc8bETlYnAynVJWa6wddaUjWismFiHSJ+/KnCSokRf6cg/Cccn4xD2H8ib/J6cfahIOK9lbNlM7puiv5VSIqXUW8E22mPh6C4SCRglN9pwFeLvgt2weQ4004N3lJHAghr0zNGGBA2D9uwYI3somqzxwIcKx50LYCPWooWjs0BMYYTvKdTeWmaaZNRr1cnQb/Gf3Fg2YdkfDUoq4W3SWypu5uQVUIe/pLDOzrwStzmju0n9rptGcJ8bKiSgHNeyQ8vZjS54OKwuNLZZZ/OIoJJs2udYFsW30QLwtV16Vm61ICuu1y/L6fSLkK5i7laKkirZ3Q2YxQjtPdBV060wq2s67fnFUGT6mUrgyMXN9CgOMtojQUDOUDO794zsZEk3KdIylebuV1+7ADPEe8rincSEXuGrgvvuDc4F+3JROWNlRgdJPdvcTUet0A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 12/04/2024 11:29, Barry Song wrote: > On Fri, Apr 12, 2024 at 10:25 PM Ryan Roberts wrote: >> >> On 12/04/2024 11:17, Barry Song wrote: >>> On Fri, Apr 12, 2024 at 9:56 PM Ryan Roberts wrote: >>>> >>>> On 12/04/2024 10:43, Barry Song wrote: >>>>> On Fri, Apr 12, 2024 at 9:27 PM Ryan Roberts wrote: >>>>>> >>>>>> Hi Barry, >>>>>> >>>>>> 2 remaining comments - otherwise looks good. (same comments I just made in the >>>>>> v4 conversation). >>>>>> >>>>>> On 12/04/2024 08:37, Barry Song wrote: >>>>>>> From: Barry Song >>>>>>> >>>>>>> Profiling a system blindly with mTHP has become challenging due to the >>>>>>> lack of visibility into its operations. Presenting the success rate of >>>>>>> mTHP allocations appears to be pressing need. >>>>>>> >>>>>>> Recently, I've been experiencing significant difficulty debugging >>>>>>> performance improvements and regressions without these figures. It's >>>>>>> crucial for us to understand the true effectiveness of mTHP in real-world >>>>>>> scenarios, especially in systems with fragmented memory. >>>>>>> >>>>>>> This patch establishes the framework for per-order mTHP >>>>>>> counters. It begins by introducing the anon_fault_alloc and >>>>>>> anon_fault_fallback counters. Additionally, to maintain consistency >>>>>>> with thp_fault_fallback_charge in /proc/vmstat, this patch also tracks >>>>>>> anon_fault_fallback_charge when mem_cgroup_charge fails for mTHP. >>>>>>> Incorporating additional counters should now be straightforward as well. >>>>>>> >>>>>>> Signed-off-by: Barry Song >>>>>>> Cc: Chris Li >>>>>>> Cc: David Hildenbrand >>>>>>> Cc: Domenico Cerasuolo >>>>>>> Cc: Kairui Song >>>>>>> Cc: Matthew Wilcox (Oracle) >>>>>>> Cc: Peter Xu >>>>>>> Cc: Ryan Roberts >>>>>>> Cc: Suren Baghdasaryan >>>>>>> Cc: Yosry Ahmed >>>>>>> Cc: Yu Zhao >>>>>>> --- >>>>>>> include/linux/huge_mm.h | 51 ++++++++++++++++++++++++++++++++++ >>>>>>> mm/huge_memory.c | 61 +++++++++++++++++++++++++++++++++++++++++ >>>>>>> mm/memory.c | 3 ++ >>>>>>> mm/page_alloc.c | 4 +++ >>>>>>> 4 files changed, 119 insertions(+) >>>>>>> >>>>>>> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h >>>>>>> index e896ca4760f6..c5beb54b97cb 100644 >>>>>>> --- a/include/linux/huge_mm.h >>>>>>> +++ b/include/linux/huge_mm.h >>>>>>> @@ -264,6 +264,57 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, >>>>>>> enforce_sysfs, orders); >>>>>>> } >>>>>>> >>>>>>> +enum mthp_stat_item { >>>>>>> + MTHP_STAT_ANON_FAULT_ALLOC, >>>>>>> + MTHP_STAT_ANON_FAULT_FALLBACK, >>>>>>> + MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, >>>>>>> + __MTHP_STAT_COUNT >>>>>>> +}; >>>>>>> + >>>>>>> +struct mthp_stat { >>>>>>> + unsigned long stats[0][__MTHP_STAT_COUNT]; >>>>>>> +}; >>>>>>> + >>>>>>> +extern struct mthp_stat __percpu *mthp_stats; >>>>>>> + >>>>>>> +static inline void count_mthp_stat(int order, enum mthp_stat_item item) >>>>>>> +{ >>>>>>> + if (order <= 0 || order > PMD_ORDER || !mthp_stats) >>>>>>> + return; >>>>>>> + >>>>>>> + this_cpu_inc(mthp_stats->stats[order][item]); >>>>>>> +} >>>>>>> + >>>>>>> +static inline void count_mthp_stats(int order, enum mthp_stat_item item, long delta) >>>>>>> +{ >>>>>>> + if (order <= 0 || order > PMD_ORDER || !mthp_stats) >>>>>>> + return; >>>>>>> + >>>>>>> + this_cpu_add(mthp_stats->stats[order][item], delta); >>>>>>> +} >>>>>>> + >>>>>>> +/* >>>>>>> + * Fold the foreign cpu mthp stats into our own. >>>>>>> + * >>>>>>> + * This is adding to the stats on one processor >>>>>>> + * but keeps the global counts constant. >>>>>>> + */ >>>>>>> +static inline void mthp_stats_fold_cpu(int cpu) >>>>>>> +{ >>>>>>> + struct mthp_stat *fold_stat; >>>>>>> + int i, j; >>>>>>> + >>>>>>> + if (!mthp_stats) >>>>>>> + return; >>>>>>> + fold_stat = per_cpu_ptr(mthp_stats, cpu); >>>>>>> + for (i = 1; i <= PMD_ORDER; i++) { >>>>>>> + for (j = 0; j < __MTHP_STAT_COUNT; j++) { >>>>>>> + count_mthp_stats(i, j, fold_stat->stats[i][j]); >>>>>>> + fold_stat->stats[i][j] = 0; >>>>>>> + } >>>>>>> + } >>>>>>> +} >>>>>> >>>>>> This is a pretty horrible hack; I'm pretty sure just summing for all *possible* >>>>>> cpus should work. >>>>>> >>>>>>> + >>>>>>> #define transparent_hugepage_use_zero_page() \ >>>>>>> (transparent_hugepage_flags & \ >>>>>>> (1<>>>>>> diff --git a/mm/huge_memory.c b/mm/huge_memory.c >>>>>>> index dc30139590e6..21c4ac74b484 100644 >>>>>>> --- a/mm/huge_memory.c >>>>>>> +++ b/mm/huge_memory.c >>>>>>> @@ -526,6 +526,50 @@ static const struct kobj_type thpsize_ktype = { >>>>>>> .sysfs_ops = &kobj_sysfs_ops, >>>>>>> }; >>>>>>> >>>>>>> +struct mthp_stat __percpu *mthp_stats; >>>>>>> + >>>>>>> +static unsigned long sum_mthp_stat(int order, enum mthp_stat_item item) >>>>>>> +{ >>>>>>> + unsigned long sum = 0; >>>>>>> + int cpu; >>>>>>> + >>>>>>> + cpus_read_lock(); >>>>>>> + for_each_online_cpu(cpu) { >>>>>>> + struct mthp_stat *this = per_cpu_ptr(mthp_stats, cpu); >>>>>>> + >>>>>>> + sum += this->stats[order][item]; >>>>>>> + } >>>>>>> + cpus_read_unlock(); >>>>>>> + >>>>>>> + return sum; >>>>>>> +} >>>>>>> + >>>>>>> +#define DEFINE_MTHP_STAT_ATTR(_name, _index) \ >>>>>>> +static ssize_t _name##_show(struct kobject *kobj, \ >>>>>>> + struct kobj_attribute *attr, char *buf) \ >>>>>>> +{ \ >>>>>>> + int order = to_thpsize(kobj)->order; \ >>>>>>> + \ >>>>>>> + return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index)); \ >>>>>>> +} \ >>>>>>> +static struct kobj_attribute _name##_attr = __ATTR_RO(_name) >>>>>>> + >>>>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC); >>>>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); >>>>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); >>>>>>> + >>>>>>> +static struct attribute *stats_attrs[] = { >>>>>>> + &anon_fault_alloc_attr.attr, >>>>>>> + &anon_fault_fallback_attr.attr, >>>>>>> + &anon_fault_fallback_charge_attr.attr, >>>>>>> + NULL, >>>>>>> +}; >>>>>>> + >>>>>>> +static struct attribute_group stats_attr_group = { >>>>>>> + .name = "stats", >>>>>>> + .attrs = stats_attrs, >>>>>>> +}; >>>>>>> + >>>>>>> static struct thpsize *thpsize_create(int order, struct kobject *parent) >>>>>>> { >>>>>>> unsigned long size = (PAGE_SIZE << order) / SZ_1K; >>>>>>> @@ -549,6 +593,12 @@ static struct thpsize *thpsize_create(int order, struct kobject *parent) >>>>>>> return ERR_PTR(ret); >>>>>>> } >>>>>>> >>>>>>> + ret = sysfs_create_group(&thpsize->kobj, &stats_attr_group); >>>>>>> + if (ret) { >>>>>>> + kobject_put(&thpsize->kobj); >>>>>>> + return ERR_PTR(ret); >>>>>>> + } >>>>>>> + >>>>>>> thpsize->order = order; >>>>>>> return thpsize; >>>>>>> } >>>>>>> @@ -691,6 +741,11 @@ static int __init hugepage_init(void) >>>>>>> */ >>>>>>> MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER < 2); >>>>>>> >>>>>>> + mthp_stats = __alloc_percpu((PMD_ORDER + 1) * sizeof(mthp_stats->stats[0]), >>>>>>> + sizeof(unsigned long)); >>>>>> >>>>>> Personally I think it would be cleaner to allocate statically using >>>>>> ilog2(MAX_PTRS_PER_PTE) instead of PMD_ORDER. >>>>> >>>>> Hi Ryan, >>>>> >>>>> I don't understand why MAX_PTRS_PER_PTE is the correct size. For ARM64, >>>>> >>>>> #define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) >>>>> >>>>> #define MAX_PTRS_PER_PTE PTRS_PER_PTE >>>>> >>>>> #define PTRS_PER_PTE (1 << (PAGE_SHIFT - 3)) >>>>> >>>>> while PAGE_SIZE is 16KiB or 64KiB, PTRS_PER_PTE can be a huge number? >>>>> >>>>> >>>>> Am I missing something? >>>> >>>> PTRS_PER_PTE is the number of PTE entries in a PTE table. On arm64 its as follows: >>>> >>>> PAGE_SIZE PAGE_SHIFT PTRS_PER_PTE >>>> 4K 12 512 >>>> 16K 14 2048 >>>> 64K 16 8192 >>>> >>>> So (PTRS_PER_PTE * PAGE_SIZE) = PMD_SIZE >>>> >>>> PMD_ORDER is ilog2(PMD_SIZE / PAGE_SIZE) = ilog2(PTRS_PER_PTE) >>>> >>>> MAX_PTRS_PER_PTE is just the maximum value that PTRS_PER_PTE will ever have, >>>> (and its equal to PTRS_PER_PTE except for powerpc). >>>> >>>> Pretty sure the math is correct? >>> >>> I am not convinced the math is correct :-) >>> >>> while page size is 64KiB, the page table is as below, >>> PMD_ORDER = L2 index bits = [41:29] = 13 != ilog2(8192) >> >> 1 << 13 = 8192 >> >> Right? So: >> >> ilog2(8192) = 13 >> >> What's wrong with that? >> >> I even checked in Python to make sure I'm not going mad: >> >>>>> import math >>>>> math.log2(8192) >> 13.0 > > You're correct. My mind fixated on the '16' in the line '64K 16 8192'. > I mistakenly thought ilog2(8192) equals 16. Apologies for the confusion. No worries! We got there in the end :) Of course my suggestion relies on being able to get a compile-time constant from ilog2(MAX_PTRS_PER_PTE). I think that should work, right? > >> >>> >>> >>> +--------+--------+--------+--------+--------+--------+--------+--------+ >>> |63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0| >>> +--------+--------+--------+--------+--------+--------+--------+--------+ >>> | | | | | >>> | | | | v >>> | | | | [15:0] in-page offset >>> | | | +----------> [28:16] L3 index >>> | | +--------------------------> [41:29] L2 index >>> | +-------------------------------> [47:42] L1 index (48-bit) >>> | [51:42] L1 index (52-bit) >>> +-------------------------------------------------> [63] TTBR0/1 >>> >>> while page size is 4KiB, the page table is as below, >>> >>> +--------+--------+--------+--------+--------+--------+--------+--------+ >>> |63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0| >>> +--------+--------+--------+--------+--------+--------+--------+--------+ >>> | | | | | | >>> | | | | | v >>> | | | | | [11:0] in-page offset >>> | | | | +-> [20:12] L3 index >>> | | | +-----------> [29:21] L2 index >>> | | +---------------------> [38:30] L1 index >>> | +-------------------------------> [47:39] L0 index >>> +-------------------------------------------------> [63] TTBR0/1 >>> >>> PMD_ORDER = L2 index bits = [29:21] = 9 = ilog2(512). >>> >>> You are only correct while page size = 4KiB. >>> >>> >>> >>> >>