From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69840C25B08 for ; Sat, 20 Aug 2022 22:08:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 775EA8D0001; Sat, 20 Aug 2022 18:08:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 724746B0074; Sat, 20 Aug 2022 18:08:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5EC348D0001; Sat, 20 Aug 2022 18:08:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4FA036B0073 for ; Sat, 20 Aug 2022 18:08:18 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 01ABA140F55 for ; Sat, 20 Aug 2022 22:08:17 +0000 (UTC) X-FDA: 79821360276.11.2571873 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf31.hostedemail.com (Postfix) with ESMTP id 6A65320034 for ; Sat, 20 Aug 2022 22:07:15 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7171C60C36; Sat, 20 Aug 2022 22:07:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B4A6DC433C1; Sat, 20 Aug 2022 22:07:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1661033233; bh=OMjSKoN1CgLtZDBS1iDToK7phTnQHbLYbNnSr9H4cOY=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=SST4XN0HIP94HTmFajkz8ZT0FtzmNfPwNmCIamX1rrN1Q3BSh2pw1B8pLTSZs/E1U 96qlzs7VEWwlTPlxQScCv5g2GCPhBGyg+4pdbRPd9G+/PyC3/AKBJuN3RbPmMfywxW YG/Z7Qz+KVE1vLZD3ZGgAtoAJeMlCw+oXB0w9NCs= Date: Sat, 20 Aug 2022 15:07:12 -0700 From: Andrew Morton To: Cc: , , Subject: Re: [PATCH v3] mm: add thp_utilization metrics to debugfs Message-Id: <20220820150712.53ec2dd281dfe894ad3fe2df@linux-foundation.org> In-Reply-To: <20220818000112.2722201-1-alexlzhu@fb.com> References: <20220818000112.2722201-1-alexlzhu@fb.com> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; x86_64-redhat-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661033235; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OBOZqvcLSiOMKGdWOkjlvtXEDtASJYCsPv0gEs2QNuI=; b=RPxBcRXBWweNtrD6+xjxm9hmyacfCvkLMuts7IuEE4Paz9LD6yPfscGwMJAtSKY72K694w ek4wUGvfxx5DlY2mDGP9ACRRO4LmsUdNPKryDc5zso5KxQmTGLA6LneJELKgSL9jrOXIcU s8icUdmHHjCR1mnId/TfnsPoqDRGX9k= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=SST4XN0H; spf=pass (imf31.hostedemail.com: domain of akpm@linux-foundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661033235; a=rsa-sha256; cv=none; b=mj4XLA3ZEAXALc+huFTkOPHhCeSzZDnlGx6drGb7T0V1B1V+hEau1ldm8dnBmuuhT45IXj I/9+Q8aFkPq7f9Q6QvgeTtp04coZNYLDA6OcjyfIGGY27U6KBXp5fbwfymnCEUgMnGLJ3R 1TZkQ3iB4JZyp1lV5K4jvsHGmmm0bcA= Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=SST4XN0H; spf=pass (imf31.hostedemail.com: domain of akpm@linux-foundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6A65320034 X-Stat-Signature: 37cqjmnd343u4qpa1esxqad3x4scqhkq X-Rspam-User: X-HE-Tag: 1661033235-77696 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 17 Aug 2022 17:01:12 -0700 wrote: > THPs have historically been enabled on a per application basis due to > performance increase or decrease depending on how the particular > application uses physical memory. When THPs are heavily utilized, > application performance improves due to fewer TLB cache misses. > It has long been suspected that performance regressions when THP > is enabled happens due to heavily underutilized anonymous THPs. > > Previously there was no way to track how much of a THP is > actually being used. With this change, we seek to gain visibility > into the utilization of THPs in order to make more intelligent > decisions regarding paging. > > This change introduces a tool that scans through all of physical > memory for anonymous THPs and groups them into buckets based > on utilization. It also includes an interface under > /sys/kernel/debug/thp_utilization. > > Utilization of a THP is defined as the percentage of nonzero > pages in the THP. The worker thread will scan through all > of physical memory and obtain utilization of all anonymous > THPs. It will gather this information by periodically scanning > through all of physical memory for anonymous THPs, group them > into buckets based on utilization, and report utilization I'd like to see sample debugfs output right here in the changelog, for reviewers to review. In some detail. And I'd like to see the code commented! Especially thp_utilization_workfn(), thp_util_scan() and thp_scan_next_zone(). What are their roles and responsibilities? How long do they take, by what means do they scan? I mean, scanning all of physical memory is a huge task. How do we avoid chewing vast amounts of CPU? What is the chosen approach and what are the tradeoffs? Why is is done within a kernel thread at all, rather than putting the load into the context of the reader of the stats (which is more appropriate). etcetera. There are many traps, tradeoffs and hidden design decisions here. Please unhide them. This comment, which is rather a core part of these tradeoffs: +/* + * The number of addresses to scan through on each periodic + * run of the scanner that generates /sys/kernel/debug/thp_utilization. + */ +#define THP_UTIL_SCAN_SIZE 256 isn't very helpful. "number of addresses"? Does it mean we scan 256 bytes at a time? 256 pages? 256 hugepages? Something else? How can any constant make sense when different architectures have different [huge]page sizes? Should it be scaled by pagesize? And if we're going to do that, we should scale it by CPU speed at the same time. Or bypass all of that and simply scan for a certain amount of *time*, rather than scan a certain amount of memory. After all, chunking up the scan time is what we're trying to achieve by chunking up the scan amount. Why not chunk up the scan time directly? See where I'm going? I see many hidden assumptions, design decisions and tradeoffs here. Can we please attempt to spell them out and review them. Anyway. There were many review comments on previous versions. It would have been better had those reviewers been cc'ed on this version. I'll go into hiding and see what people think.