From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA32DC4345F for ; Fri, 12 Apr 2024 11:59:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 437A96B009C; Fri, 12 Apr 2024 07:59:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E7EA6B009D; Fri, 12 Apr 2024 07:59:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FDC56B009E; Fri, 12 Apr 2024 07:59:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 122CC6B009C for ; Fri, 12 Apr 2024 07:59:19 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C2B8D140E86 for ; Fri, 12 Apr 2024 11:59:18 +0000 (UTC) X-FDA: 82000734396.12.FDD6BDA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf26.hostedemail.com (Postfix) with ESMTP id 9A54514000C for ; Fri, 12 Apr 2024 11:59:16 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712923157; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tiBaR15wHfnOYmjpQ3sv7KT15d4IJQ2MLx/w2QNgr1g=; b=PW+OZNg+c92AxE9E/SZE1NmPuXnlE6VHxG154R3bWbXcJMuwb3tlBB+ItSPWItZU5329y0 0EanNMGagi1HowOAP5hfyEzNxHIcz+nJ9KI7nHwb0pft+Cr9od0sx7ulMlyH4PpHSH8Nfc c5c46iS+XZ+3YBOhbirQwMFJ7vu3rQc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712923157; a=rsa-sha256; cv=none; b=LxoiQKbPSWUCSwtzBNKxYM2c/Plt2YCWpqha2KpeT71xOlr3YfuDvoYoeZ0q/dcKYwi3cx GTBJlopY7ac+XxiUxYUTOm/g+p1ZxDY27oD9O2u5aBjLaqYfMKkMXrDvh0Vi7guiElUv67 Mbpkm8GL73wAbw6SghefYgKYbZQfiGU= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; spf=pass (imf26.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ECFEE339; Fri, 12 Apr 2024 04:59:44 -0700 (PDT) Received: from [10.57.73.208] (unknown [10.57.73.208]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 80C613F766; Fri, 12 Apr 2024 04:59:13 -0700 (PDT) Message-ID: Date: Fri, 12 Apr 2024 12:59:11 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v6 1/4] mm: add per-order mTHP anon_fault_alloc and anon_fault_fallback counters Content-Language: en-GB To: Barry Song <21cnbao@gmail.com>, akpm@linux-foundation.org, linux-mm@kvack.org Cc: cerasuolodomenico@gmail.com, chrisl@kernel.org, david@redhat.com, kasong@tencent.com, linux-kernel@vger.kernel.org, peterx@redhat.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, yosryahmed@google.com, yuzhao@google.com, corbet@lwn.net References: <20240412114858.407208-1-21cnbao@gmail.com> <20240412114858.407208-2-21cnbao@gmail.com> From: Ryan Roberts In-Reply-To: <20240412114858.407208-2-21cnbao@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Stat-Signature: kqnfgrcq3tctyzmyip9pwm7demjxz8b1 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 9A54514000C X-Rspam-User: X-HE-Tag: 1712923156-142274 X-HE-Meta: U2FsdGVkX1+ovDI53teZM8sbfE8N0m+6pbvGTru58+FhOAwpRPy9ELcVo2eNJODW4UPSOdHZaDnDeE5Adl6SXm5oHSEEF1LH86K/8b2+gkKKOU3tRfL8DhFPpE+qHEfgl3rQ6heOCTs/JtsL5st4NanTC0qqzsXd78iTaTGUg3dPkQujPb4p7/D8Nf8DJkmHw3ygRE+Rj8AnaxMHfW2g16tn8gHf3STiGXMB1JpHpJUoZR7ZFTWMHoa+O71Ne2yKi4VeHPFm1c1BJFPqQMKM2ODDAGj8B92e3UpRGV60P45IMb3c7IHuNiVBGzP/3iMmxeQmZQAxmuB8yfLg2lVwduHXG8fHcLYaqXEVhcwVRs7u6XpWEE4aiKvw2ZkvMprOn8qHkZ/9JuWJZhmMKPcxD9++VTQ5tX8q3LcbTNSJ9T/xsJitmBsVo/d3uG9FfBuGGLwPftxxMolWoFoagl8aT1IAR0TTu4kPfveLQrHwX17KyAKdPBzwGKowVfZVU4Hei2hcETJ4mIH+oTdMKsJA/VaUpHaXyE5KUf1TifP2yuM7qt4derZENZJFqNQA43faojq+bVhDO4ayDrSDtX+C4RRB7MPJ2GCsX9uzbI52tWcpZssIsAGVqVkOLmhzvsZsbnrJKEu1kPlCIxz6aUXFu5gm5a+dPcMfSRcBm74DWA+GnUeCJp8vJvSqHbvb9uKgt7XoXLkkVegI2pUrFFQAhMKr8W6NdKThXGgOKi8WXBmjigTDsbD+qBCw4dYwV0jMLyj0nb8APG11D6k+YJoIjcdIRYA2dboJenenW/XMUpHpcIbCOecbD3rmhr6QcL0jejxrJF5nX+wvfjmbB5HF8H/lrjzB2z4ycCpFhXVg2jaHCdgVBFR+CbSulHnLK3gm1PpxpHQW+qPiqlgmObdcBS6N6r7yEVoAvMhDpSOfdUcbC3XRhc/YMtqPNM49K6a9tgs7KN0myAadlZU5TaM Iycx2IQX BVrA1QB7UxpitVu30X+7q5i084+XEdcyEB70N+cOuRHGLdra8OnpwwmgDuaAmqaJ4w1CBHHRXvU5dqlo4SV8DEjon24e9KQ+3ClF40+XJ3P7cbUWTt1HEgvZwAdUOFf0wJQ7+xLk16JiUV2PKRPqAN3i1oJjZVO6eCYpT/z0KMSrAC4y6Yq5emamURCnAVygVoy4+kirCFfq3fyYhDCam94rLJnHRSg0HoLzyXGkOXRxSRCzYEMx9ryVtiH38oykf1SfkwyTIpYoHdvxs+srpcD6U5qreP3aEdHpyRfOJSQuQTAfvMugCZEeGLZ85XH3fyCfRV52xV65Xu+umzhG8R12sx4xWAYWNV14HtYyBDKYIjrg0WnpqORWSuJMRsZcnsXo9VAGLMC+MSoZMw8QLmnl7dk0K7UC6t8gnK6SKDVuTwk/wtYpKMqTz2OPcPqLBxvDH/+K3qihF7OSOUQuU4eCvNMtDedCMIPCxyNj1dAz0nUeMOkcsi3h3hpbxvtc+JeW60oubEz56y0uM9ns8wWEzE3+ML7Q3QslQe/9RWViz55yNGPBrPyHTPw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 12/04/2024 12:48, Barry Song wrote: > From: Barry Song > > Profiling a system blindly with mTHP has become challenging due to the > lack of visibility into its operations. Presenting the success rate of > mTHP allocations appears to be pressing need. > > Recently, I've been experiencing significant difficulty debugging > performance improvements and regressions without these figures. It's > crucial for us to understand the true effectiveness of mTHP in real-world > scenarios, especially in systems with fragmented memory. > > This patch establishes the framework for per-order mTHP > counters. It begins by introducing the anon_fault_alloc and > anon_fault_fallback counters. Additionally, to maintain consistency > with thp_fault_fallback_charge in /proc/vmstat, this patch also tracks > anon_fault_fallback_charge when mem_cgroup_charge fails for mTHP. > Incorporating additional counters should now be straightforward as well. > > Signed-off-by: Barry Song > Cc: Chris Li > Cc: David Hildenbrand > Cc: Domenico Cerasuolo > Cc: Kairui Song > Cc: Matthew Wilcox (Oracle) > Cc: Peter Xu > Cc: Ryan Roberts > Cc: Suren Baghdasaryan > Cc: Yosry Ahmed > Cc: Yu Zhao LGTM! Reviewed-by: Ryan Roberts > --- > include/linux/huge_mm.h | 21 +++++++++++++++++ > mm/huge_memory.c | 52 +++++++++++++++++++++++++++++++++++++++++ > mm/memory.c | 5 ++++ > 3 files changed, 78 insertions(+) > > diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > index e896ca4760f6..d4fdb2641070 100644 > --- a/include/linux/huge_mm.h > +++ b/include/linux/huge_mm.h > @@ -264,6 +264,27 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, > enforce_sysfs, orders); > } > > +enum mthp_stat_item { > + MTHP_STAT_ANON_FAULT_ALLOC, > + MTHP_STAT_ANON_FAULT_FALLBACK, > + MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, > + __MTHP_STAT_COUNT > +}; > + > +struct mthp_stat { > + unsigned long stats[ilog2(MAX_PTRS_PER_PTE) + 1][__MTHP_STAT_COUNT]; > +}; > + > +DECLARE_PER_CPU(struct mthp_stat, mthp_stats); > + > +static inline void count_mthp_stat(int order, enum mthp_stat_item item) > +{ > + if (order <= 0 || order > PMD_ORDER) > + return; > + > + this_cpu_inc(mthp_stats.stats[order][item]); > +} > + > #define transparent_hugepage_use_zero_page() \ > (transparent_hugepage_flags & \ > (1< diff --git a/mm/huge_memory.c b/mm/huge_memory.c > index dc30139590e6..dfc38cc83a04 100644 > --- a/mm/huge_memory.c > +++ b/mm/huge_memory.c > @@ -526,6 +526,48 @@ static const struct kobj_type thpsize_ktype = { > .sysfs_ops = &kobj_sysfs_ops, > }; > > +DEFINE_PER_CPU(struct mthp_stat, mthp_stats) = {{{0}}}; > + > +static unsigned long sum_mthp_stat(int order, enum mthp_stat_item item) > +{ > + unsigned long sum = 0; > + int cpu; > + > + for_each_possible_cpu(cpu) { > + struct mthp_stat *this = &per_cpu(mthp_stats, cpu); > + > + sum += this->stats[order][item]; > + } > + > + return sum; > +} > + > +#define DEFINE_MTHP_STAT_ATTR(_name, _index) \ > +static ssize_t _name##_show(struct kobject *kobj, \ > + struct kobj_attribute *attr, char *buf) \ > +{ \ > + int order = to_thpsize(kobj)->order; \ > + \ > + return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index)); \ > +} \ > +static struct kobj_attribute _name##_attr = __ATTR_RO(_name) > + > +DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC); > +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); > +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); > + > +static struct attribute *stats_attrs[] = { > + &anon_fault_alloc_attr.attr, > + &anon_fault_fallback_attr.attr, > + &anon_fault_fallback_charge_attr.attr, > + NULL, > +}; > + > +static struct attribute_group stats_attr_group = { > + .name = "stats", > + .attrs = stats_attrs, > +}; > + > static struct thpsize *thpsize_create(int order, struct kobject *parent) > { > unsigned long size = (PAGE_SIZE << order) / SZ_1K; > @@ -549,6 +591,12 @@ static struct thpsize *thpsize_create(int order, struct kobject *parent) > return ERR_PTR(ret); > } > > + ret = sysfs_create_group(&thpsize->kobj, &stats_attr_group); > + if (ret) { > + kobject_put(&thpsize->kobj); > + return ERR_PTR(ret); > + } > + > thpsize->order = order; > return thpsize; > } > @@ -880,6 +928,8 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, > folio_put(folio); > count_vm_event(THP_FAULT_FALLBACK); > count_vm_event(THP_FAULT_FALLBACK_CHARGE); > + count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_FALLBACK); > + count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); > return VM_FAULT_FALLBACK; > } > folio_throttle_swaprate(folio, gfp); > @@ -929,6 +979,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, > mm_inc_nr_ptes(vma->vm_mm); > spin_unlock(vmf->ptl); > count_vm_event(THP_FAULT_ALLOC); > + count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_ALLOC); > count_memcg_event_mm(vma->vm_mm, THP_FAULT_ALLOC); > } > > @@ -1050,6 +1101,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) > folio = vma_alloc_folio(gfp, HPAGE_PMD_ORDER, vma, haddr, true); > if (unlikely(!folio)) { > count_vm_event(THP_FAULT_FALLBACK); > + count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_FALLBACK); > return VM_FAULT_FALLBACK; > } > return __do_huge_pmd_anonymous_page(vmf, &folio->page, gfp); > diff --git a/mm/memory.c b/mm/memory.c > index 649a547fe8e3..f31da2de19c6 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -4368,6 +4368,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) > folio = vma_alloc_folio(gfp, order, vma, addr, true); > if (folio) { > if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) { > + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); > folio_put(folio); > goto next; > } > @@ -4376,6 +4377,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) > return folio; > } > next: > + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK); > order = next_order(&orders, order); > } > > @@ -4485,6 +4487,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) > > folio_ref_add(folio, nr_pages - 1); > add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE > + count_mthp_stat(folio_order(folio), MTHP_STAT_ANON_FAULT_ALLOC); > +#endif > folio_add_new_anon_rmap(folio, vma, addr); > folio_add_lru_vma(folio, vma); > setpte: