From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36D4DC54E67 for ; Wed, 27 Mar 2024 12:17:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9D8096B0085; Wed, 27 Mar 2024 08:17:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9890B6B0089; Wed, 27 Mar 2024 08:17:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8765C6B008A; Wed, 27 Mar 2024 08:17:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 69BFF6B0085 for ; Wed, 27 Mar 2024 08:17:30 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0BA28C0BA5 for ; Wed, 27 Mar 2024 12:17:30 +0000 (UTC) X-FDA: 81942719460.01.CFFD38B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf03.hostedemail.com (Postfix) with ESMTP id E09982001D for ; Wed, 27 Mar 2024 12:17:27 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711541848; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SUI6Xk927WLmOSiT80LqNRG4YGN6f5KRH0Lm0JUqWhU=; b=MuU45nD9sL8oD/MhoGtNfA4btI2xjD/A5lE31fnUKX/EaOAeDWW/5kHXRgvvua+K+lmWtK Q+osrPvzzX87K+iAZz1HNi9V8m0NeZjqALvVhkOjFK6BkNKTVxphci1TKHBlD/Va7PGFR4 yw5Yb27lKhMfGxEDEd+MJVGyQ4cqPuo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711541848; a=rsa-sha256; cv=none; b=yeiUoY13bR822Z07OzHSdIl80rP0nPx0affE7qSSdgD2MxeOU63FquzLlsolPSBt1zdxyR CC0f8VGAJsExqm97otWdCx9tsmfUPlq8etnJGJP340Mb44AFlx4p3df+/uZbsKDvyeKeBc dPw+IBQLyuM01ydtwY+QNfp4I8Kmieo= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CFDD12F4; Wed, 27 Mar 2024 05:18:00 -0700 (PDT) Received: from [10.57.72.121] (unknown [10.57.72.121]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A29183F694; Wed, 27 Mar 2024 05:17:24 -0700 (PDT) Message-ID: <8a051c7c-a9a7-4967-bfc3-224a93e7afcf@arm.com> Date: Wed, 27 Mar 2024 12:17:22 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] mm: show mthp_fault_alloc and mthp_fault_fallback of multi-size THPs Content-Language: en-GB To: David Hildenbrand , Barry Song <21cnbao@gmail.com>, Matthew Wilcox , yuzhao@google.com Cc: akpm@linux-foundation.org, linux-mm@kvack.org, kasong@tencent.com, yosryahmed@google.com, cerasuolodomenico@gmail.com, surenb@google.com, Barry Song , Johannes Weiner References: <20240326030103.50678-1-21cnbao@gmail.com> <330c90c6-87ef-4497-a9c7-739dcd686ca2@redhat.com> From: Ryan Roberts In-Reply-To: <330c90c6-87ef-4497-a9c7-739dcd686ca2@redhat.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: E09982001D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: jax54359cx41tk7krbk8q6soesnqwwyb X-HE-Tag: 1711541847-260882 X-HE-Meta: U2FsdGVkX18Lugodj20meTabIpv4WChucqBcG4zagosMlBNq0fdtirsNbhn/yoAffprpgTRGwXENON9gLLdFJ2Bm3MtyNeRwD4TdNrGRMV4MIEiOjwyzzUwYOi1hWN+LICRiQhIZUYkRLUGt0NQ0KMq2SR04Pk+7nThPYc/FIE4jZle9r+am8qTdj2ubuyp9ow0Lqjz8I3UQ7iUuXrNZL45GDjdxxq4nswwQq/UtIGCYCa/dLHkMLkdZ+VPI9ycuJtsGbf+PI1hARMdjyWXSDXaN32osjMtmS0RFhADjHRE8qyWVN5xXqgRohEtmVw/PWGk6atLVNXZ9NbupqXgnfiy6XCSfMfFCPa3cfh0ZdY3Rnd5E9IQcgnBTkfKtNm8AKfl/LOLld7s99FU7f43ci9a5XshwJHolGowWj9olMhg9GqMG3CLLGuuGNjpxStJQigU6h5y/BA0/pWurTGXedtnEUsUil8jkp3zDW40ffriz9f7p0j2gPRmmoF/cT8X00c3IfCnF7hZgFQMDwMPHyAprm2l0tOUy6BNN6d8UwhEeGuS+dUtqTDr5lf4edTi80cFsPdXX52ksHGMtpk2XNkWvSN/9JJqlJNScI+g05SeOGkKfwoQBey8xJYj1fZWaex1/TcJXjxwVmNYXdYg4MUMuSEcWmRKz043sY+mhqFAJ97giWKiv4wEF1EeyVucXhiF0oiHvke1N2bi9RUY3zf+NT8I/R8kVZGFUzEIHe5EgaaxZlOLz9JsfUh84IckWilUtnnMWq3fi1QftZdIjLG0uXRd8yfktv8A6Udy9Crmh36H9gBsLE20q5VN1pTX5+UBsiz1HcsukO8E6TqGLz2XHMCHMXPYtfaTB2Tb+SMQuza0pRFS6EbMcB16gmJvUa7AZjqtj4kz5Hta98/SsFFoD7DDSpjbUR3F+16S7Ed3C+mI0qEpGE5pqUWJGGYPHdhytQQT13hoF12T6v7h icbGqzqd UKfM1UNte0bAMRUBNB/Hi0/gaEcbvhw5wXVYCgsG4F99ThrOn9U6E0eNNyXOjvq6V3JvKRSHOrCzD1QU5g89DIy2Dq7un/8elMic2rwLppHEZy7dp5oUMUVVvNTWwaxXZxfHmaJjmEvDAqLWs+hs0kBd8+ohv1lL3SrkZr1/qOqdnUdfoSUj5ZGhKhZemG2eAcLTNl2lnz5dVdPdbhCRyqip1I220D0SMyP60qkH6g+W741tWcwqg+v2HrIVkj7UDSJr9phED/avqN7aLuiI9OS/V+jatrYg+14Y95d/AiGgzTeIH44H7zHBPY/GEUhA8v+YJmqcye3dwEPMJbGakLYfOHgsnsZLDBKMHFin6xxD6yLXnlOt1AYONq5kggZdo6S04SeSkqFOx0MOTHDHc/lhWQUcbawHnIRJakPkg6297jSI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 27/03/2024 11:35, David Hildenbrand wrote: > On 26.03.24 23:19, Barry Song wrote: >> On Tue, Mar 26, 2024 at 4:40 PM Barry Song <21cnbao@gmail.com> wrote: >>> >>> On Tue, Mar 26, 2024 at 4:25 PM Matthew Wilcox wrote: >>>> >>>> On Tue, Mar 26, 2024 at 04:01:03PM +1300, Barry Song wrote: >>>>> Profiling a system blindly with mTHP has become challenging due >>>>> to the lack of visibility into its operations. While displaying >>>>> additional statistics such as partial map/unmap actions may >>>>> spark debate, presenting the success rate of mTHP allocations >>>>> appears to be a straightforward and pressing need. >>>> >>>> Ummm ... no?  Not like this anyway.  It has the bad assumption that >>>> "mTHP" only comes in one size. >>> >>> >>> I had initially considered per-size allocation and fallback before sending >>> the RFC. However, in order to prompt discussion and exploration >>> into profiling possibilities, I opted to send the simplest code instead. >>> >>> We could consider two options for displaying per-size statistics. >>> >>> 1. A single file could be used to display data for all sizes. >>> 1024KiB fault allocation: >>> 1024KiB fault fallback: >>> 512KiB fault allocation: >>> 512KiB fault fallback: >>> .... >>> 64KiB fault allocation: >>> 64KiB fault fallback: >>> >>> 2. A separate file for each size >>> For example, >>> >>> /sys/kernel/debug/transparent_hugepage/hugepages-1024kB/vmstat >>> /sys/kernel/debug/transparent_hugepage/hugepages-512kB/vmstat >>> ... >>> /sys/kernel/debug/transparent_hugepage/hugepages-64kB/vmstat >>> >> >> Hi Ryan, David, Willy, Yu, > > Hi! > >> >> I'm collecting feedback on whether you'd prefer access to something similar >> to /sys/kernel/debug/transparent_hugepage/hugepages-/stat to help >> determine the direction to take for this patch. > > I discussed in the past that we might want to place statistics into sysfs. The > idea was to place them into our new hierarchy: > > /sys/kernel/mm/transparent_hugepage/hugepages-1024kB/... > > following the "one value per file" sysfs design principle. > > We could have a new folder "stats" in there that contains files with statistics > we care about. > > Of course, we could also place that initially into debugfs in a similar fashion, > and move it over once the interface is considered good and stable. > > My 2 cents would be to avoid a "single file". Yes I agree with this. We discussed in the past and I summarised the outcome here: https://lore.kernel.org/linux-mm/6cc7d781-884f-4d8f-a175-8609732b87eb@arm.com/ There are more counters on my list than what you are proposing here. But conclusion was that they should be per-size and exposed through sysfs as David suggests. Personally I think those counters are obviously needed so prefer to go straight to sysfs if we can :) I had a low priority todo item to look at this - very pleased that you're taking it on! Copy/pasting my original summary here, for the lazy: --8<---- I just want to try to summarise the counters we have discussed in this thread to check my understanding: 1. global mTHP successful allocation counter, per mTHP size (inc only) 2. global mTHP failed allocation counter, per mTHP size (inc only) 3. global mTHP currently allocated counter, per mTHP size (inc and dec) 4. global "mTHP became partially mapped 1 or more processes" counter (inc only) I geuss the above should apply to both page cache and anon? Do we want separate counters for each? I'm not sure if we would want 4. to be per mTHP or a single counter for all? Probably the former if it provides a bit more info for neglegable cost. Where should these be exposed? I guess /proc/vmstats is the obvious place, but I don't think there is any precident for per-size counters (especially where the sizes will change depending on system). Perhaps it would be better to expose them in their per-size directories in /sys/kernel/mm/transparent_hugepage/hugepages-kB ? Additional to above global counters, there is a case for adding a per-process version of 4. to smaps to consider. --8<---- Thanks, Ryan > >> >> This is important to us because we're keen on understanding how often >> folios allocations fail on a system with limited memory, such as a phone. >> >> Presently, I've observed a success rate of under 8% for 64KiB allocations. >> Yet, integrating Yu's TAO optimization [1] and establishing an 800MiB >> nomerge zone on a phone with 8GiB memory, there's a substantial >> enhancement in the success rate, reaching approximately 40%. I'm still >> fine-tuning the optimal size for the zone. > > Just as a side note: > > I didn't have the capacity to comment in detail on the "new zones" proposal > in-depth so far (I'm hoping / assume there will be discussions at LSF/MM), but > I'm hoping we can avoid that for now and instead improve our pageblock > infrastructure, like Johannes is trying to, to achieve similar gains. > > I suspect "some things we can do with new zones we can also do with pageblocks > inside a zone". For example, there were discussions in the past to have "sticky > movable" pageblocks: pageblocks that may only contain movable data. One could do > the same with "pageblocks may not contain allocations < order X" etc. So one > could similarly optimize the memmap to some degree for these pageblocks. > > IMHO we should first try making THP <= pageblock allocations more reliable, not > using new zones, and I'm happy that Johannes et al. are doing work in that > direction. But it's a longer discussion to be had at LSF/MM. >