From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE9C5CD1288 for ; Mon, 1 Apr 2024 20:40:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36D856B0095; Mon, 1 Apr 2024 16:40:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 31BF16B0098; Mon, 1 Apr 2024 16:40:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20B9F6B0099; Mon, 1 Apr 2024 16:40:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id F1C826B0095 for ; Mon, 1 Apr 2024 16:40:56 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8F58C1A025E for ; Mon, 1 Apr 2024 20:40:56 +0000 (UTC) X-FDA: 81962132112.22.4F2F21A Received: from mail-vk1-f181.google.com (mail-vk1-f181.google.com [209.85.221.181]) by imf22.hostedemail.com (Postfix) with ESMTP id C2751C001C for ; Mon, 1 Apr 2024 20:40:53 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LzOxgAbc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712004053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/cHapvrvpXHzrohASPIaI05rzv7HPyfjyZhN2N5A4iQ=; b=5izXsMiiUshp3QgdbPX+G0kEzw5tSYiUIyxUt/NWS4DcGNEYR/WL+S5WbwzCG0w23+Af46 3Mpoql90ptQtSr90cO8zhO2Gu1yFu0LZTV61ZY0+Y0IxTM6vGlKf6YiuvT1LeNayHQ3ANk 5wJeIti/t3oOfoRDNKm94KH/SPEV9SY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LzOxgAbc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712004053; a=rsa-sha256; cv=none; b=5gRFij941HK0IYrZ7JCRWB/Tp4PdesTKUNO2hg1FNQ/CeDxw/N0R6v/XjKq4ueDma2nGnZ tiymEC/YeqA/QiQrfRTBXDu4nWhVjrZXHaApdft+31kejE1k5l+shqYTqkpRIeACGRqob5 Ny6kbgOueJ68Q/4J0C2PY6cm/+bXMro= Received: by mail-vk1-f181.google.com with SMTP id 71dfb90a1353d-4d89515ec9aso1502943e0c.1 for ; Mon, 01 Apr 2024 13:40:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712004053; x=1712608853; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=/cHapvrvpXHzrohASPIaI05rzv7HPyfjyZhN2N5A4iQ=; b=LzOxgAbc5x4yeWHSz/jyeZfP7wR3spdgs11Gv5+xso0KNPgu+1Zb2fJ7XEBM29Vzir ScFkGQ7RuJRRW9BquJfzS1C4r2UruY8giFOnAj2FXq1UEowjAPnaRSqdewx9OejaofiF xWAyLCCiF4Td3i8Vck8HoUpbRl5SU+GlHG9i9jL/Mc+G7iCsfUiF8HDskB5LNlqsXPjB A9b0yl20xTF9IprUnqAtf5q6nZ5l8b4+misLha36DdQAGJhXUJW3UkXXS49qjAexVz8i ChfwWF+QT1HeY0MVGPrYli2wto+n2MWJ1t9O1Z0iYG+Z6TOnexmSuQAZxt5a9mlGjss7 //Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712004053; x=1712608853; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/cHapvrvpXHzrohASPIaI05rzv7HPyfjyZhN2N5A4iQ=; b=d2T52aBlDWTbaN3pxbMpgNtobvGcUPYqDc8eYyElk5O/BVgMQKDUshjJ4ocbp5HyB9 9DnfVUKoq5PAQRe93E1fIokLznRKfy5CbU2m5oT/lsmTXYzywej0YLk3Tixze5kISgUw Jq/lN1QS4zxSXwwQjYMZDwqUhXpU+hwr7VO8lSTSpoTt8wPG4+t0rKr8/hqsj1iM4Nws rEMjUdv7CqeejMjG3SbpB+XAJGP78pNj15BiXKXaxYTkPhv/4ZwW+wsbBF/+H+bvzLG1 DQKXfKYSeqwEf+ExP9ofjsDV5XTqTFkbbQF7oNIh7aewx9HjuK2bRNrZvrifUyKuv9j9 5d4g== X-Forwarded-Encrypted: i=1; AJvYcCVGnAtp9bDebHq+IX+EWTSDg8wgTY/qJ+aWYksZEDHq8PwOIn4xV3EyA6SlBRJ1J+1KT6HBn/Oy7Bigk1RoQS85Pgg= X-Gm-Message-State: AOJu0YxQfovfcfywVvva3KA6TsCUTmPWPGHlRthot7MXNIL604npdk3M CfsV3SMtf+eJpQ7amCyITCVhztC9fdlZ2cvGy+d5r7x5GFsyGqTv2meoBVWKxcGcjnqNeJ5Li/N c/8huLwfZESn3X1qj+mCUJQ9j9NM= X-Google-Smtp-Source: AGHT+IHDwH58KMYK40znHEvP+wu6an8thb2EipI6537pF4ICktct0ncedEF0q8h2WJeZ8tvQ1IPmT8l0ofeDupK2A3A= X-Received: by 2002:a67:fe95:0:b0:475:ffc3:3b3e with SMTP id b21-20020a67fe95000000b00475ffc33b3emr7996174vsr.29.1712004052893; Mon, 01 Apr 2024 13:40:52 -0700 (PDT) MIME-Version: 1.0 References: <20240328095139.143374-1-21cnbao@gmail.com> In-Reply-To: From: Barry Song <21cnbao@gmail.com> Date: Tue, 2 Apr 2024 09:40:41 +1300 Message-ID: Subject: Re: [PATCH v2] mm: add per-order mTHP alloc_success and alloc_fail counters To: Yu Zhao Cc: akpm@linux-foundation.org, linux-mm@kvack.org, willy@infradead.org, david@redhat.com, ryan.roberts@arm.com, cerasuolodomenico@gmail.com, kasong@tencent.com, surenb@google.com, v-songbaohua@oppo.com, yosryahmed@google.com, chrisl@kernel.org, peterx@redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: C2751C001C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: x9awqaxc4gwug6myxawztcrsqtmgt44b X-HE-Tag: 1712004053-742642 X-HE-Meta: U2FsdGVkX19GcV0q0Vos7PJa6c+6W5N2tUsYg1uFVXKmjGziQoenuMiNZoRGaWkRa2fAqlvI7pP/yXBWWoV6qMcVTv+EkHcCtDLZAkk/Cvxjmw2BKnDBwVKwYrHv/91oCDT7LO3esmx2Y5TFnYMvb7/55dUWWBy1nmeCjvMFsSJRGT2HDBk6wyruHb2gQ/m6moAVo/WKC+eAWahnZc6OSAEHB3e4qw8xaKFhw4LtMJ5d1BkH/dGdZ/TvB1diSRryAdi1migcFPIx+zOvwxKuxxyzCFkazTri/XTfO66AnspO+5CdFl2QCLhshloDvkHHmdeoJi/TIfIwVQOcxBoV2Y8aO8YAnJVc3rXwcwq/sqE7azyHJhmMF4kmn0sHATE/95UdUpFyXalDSfJeRL3TN0NXIacPjPWwKwcx6UrtFDs8gdMoJEBKeFM8l9DUoSDYE5DDdURbg0JoherI4SCPYmbD8+o9KKdx5GeF2saRJRa9wrdNUH9V53uN6ncGePrTZuTndQj5UEEhBJy6svpu3pSpupiLHOdki5vR1BIKj4vYVmfvef1lbOM1YZG/jtt0eVQZmZUXdXESgZ1WBXFdzY4DvJHjtXLzCYDeHtkTj3QWLHaINtZh24syMTwpsv65VZmZfp88gEVL2QtxVWK7UyBMMHSsvctn60N1BhanelAfkefrzMKeyILIr5ZiRPDVNYpryScLCIEMmgxR9tXU7NGqKen5SE/1Z6tPsBaNGkOjiBlcmfzshaV8TMAfSwxvUPyIx1DBVQoe1a0pmt2icHgiXdcLnUCUntz+NMLaBPwX/H0cOWtbxWJ2wORwue6nMzkNbPyhL/3YShOTXqR7oY6JPX5h/RD4D2Q0AbgcJxHMmhhwk2OLoPjdrCEcCagLC0UKLKgpxkJu/N9Di+kvySxq6EF5KgGWogLSg3UDh4BizqJV+dXuWpddM7QNHgl7Vyds0qnPWp41TIP66g4 O5j0ZnoD XKIoPogg3jiNBu+WpeWLzjmf9vxnNZrEOSRBjKMWhqOAAVWJ21jLulga47i/vAjUtbhdzeZPo61EZV4DA0d/P95315J5hdSi5icnMeSU+MBavSZ/HomAsRD3cjULuOrp5CW8yvP09axRBmWQY+gGSBohhROyQSB3vrX8X6Z8i9Yd1RHhgzyJZif2ianKViQ/VhhPCWAqDv24XwOaxT/4ouHCZIW34X+EZ17UKxKb87qPcdMFjziB+AVeSCMwqipdFGGRv1vMg+QGj4TjF/+xhO1cFo3MTv/kPXFONVLe7/fQjaniZXihbYCZLIGsgB08nrXHxI6fVhu81K1haTCQosF5ueJMKgkscA+4bAQS0j5/ucz3/ckKBVf162sIcnN8xx7iRym567q91P/QBghZD04bOIDKIWsrDBrl2RaNb2pNFu7h3Zmn2Z0FARhiwHyS6F2sWP682/qMdz0Q7nYDK/qm5jeJaX/6GE9TkWOZUCEMTxQZur6kjjvi919SJykVvsgpdjpsjlTOwR7TTddLtZ5kNQYfvWnt3v+Bf+ytvhN8HCa6hZLKVyHgZ9aE4k8Zo6jts X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Apr 2, 2024 at 3:46=E2=80=AFAM Yu Zhao wrote: > > On Thu, Mar 28, 2024 at 5:51=E2=80=AFAM Barry Song <21cnbao@gmail.com> wr= ote: > > > > From: Barry Song > > > > Profiling a system blindly with mTHP has become challenging due > > to the lack of visibility into its operations. Presenting the > > success rate of mTHP allocations appears to be pressing need. > > > > Recently, I've been experiencing significant difficulty debugging > > performance improvements and regressions without these figures. > > It's crucial for us to understand the true effectiveness of > > mTHP in real-world scenarios, especially in systems with > > fragmented memory. > > > > This patch sets up the framework for per-order mTHP counters, > > starting with the introduction of alloc_success and alloc_fail > > counters. Incorporating additional counters should now be > > straightforward as well. > > > > The initial two unsigned longs for each event are unused, given > > that order-0 and order-1 are not mTHP. Nonetheless, this refinement > > improves code clarity. > > > > Signed-off-by: Barry Song > > --- > > -v2: > > * move to sysfs and provide per-order counters; David, Ryan, Willy > > -v1: > > https://lore.kernel.org/linux-mm/20240326030103.50678-1-21cnbao@gmail.= com/ > > > > include/linux/huge_mm.h | 17 +++++++++++++ > > mm/huge_memory.c | 54 +++++++++++++++++++++++++++++++++++++++++ > > mm/memory.c | 3 +++ > > 3 files changed, 74 insertions(+) > > > > diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > > index e896ca4760f6..27fa26a22a8f 100644 > > --- a/include/linux/huge_mm.h > > +++ b/include/linux/huge_mm.h > > @@ -264,6 +264,23 @@ unsigned long thp_vma_allowable_orders(struct vm_a= rea_struct *vma, > > enforce_sysfs, orders); > > } > > > > +enum thp_event_item { > > + THP_ALLOC_SUCCESS, > > + THP_ALLOC_FAIL, > > + NR_THP_EVENT_ITEMS > > +}; > > + > > +struct thp_event_state { > > + unsigned long event[PMD_ORDER + 1][NR_THP_EVENT_ITEMS]; > > +}; > > + > > +DECLARE_PER_CPU(struct thp_event_state, thp_event_states); > > Do we have existing per-CPU counters that cover all possible THP > orders? I.e., foo_counter[PMD_ORDER + 1][BAR_ITEMS]. I don't think we > do but I want to double check. Right. The current counters are tailored for PMD-mapped THP within the vm_event_state. Therefore, it appears that we lack counters specific to each order. > > This might be fine if BAR_ITEMS is global, not per memcg. Otherwise on > larger systems, e.g., 512 CPUs which is not uncommon, we'd have high > per-CPU memory overhead. For Google's datacenters, per-CPU memory > overhead has been a problem. Right. I don't strongly feel the need for per-memcg counters, and the /sys/kernel/mm/transparent_hugepage/hugepages- is also global. > > I'm not against this patch since NR_THP_EVENT_ITEMS is not per memcg. > Alternatively, we could make the per-CPU counters to track only one > order and flush the local counter to a global atomic counter if the > new order doesn't match the existing order stored in the local > counter. WDYT? The code assumes the worst-case scenario where users might enable multiple orders. Therefore, we require a lightweight approach to prevent frequent flushing of atomic operations. Thanks Barry