From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9CCDC38A2D for ; Wed, 26 Oct 2022 20:51:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1017E8E0002; Wed, 26 Oct 2022 16:51:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B2638E0001; Wed, 26 Oct 2022 16:51:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBB848E0002; Wed, 26 Oct 2022 16:51:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DBBA48E0001 for ; Wed, 26 Oct 2022 16:51:44 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8A6AF141187 for ; Wed, 26 Oct 2022 20:51:44 +0000 (UTC) X-FDA: 80064296928.01.726D3DC Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf27.hostedemail.com (Postfix) with ESMTP id 2512F40008 for ; Wed, 26 Oct 2022 20:51:43 +0000 (UTC) Received: by mail-pl1-f174.google.com with SMTP id p3so14436542pld.10 for ; Wed, 26 Oct 2022 13:51:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=YeygMNC1bvvP4dcp+sNmGP0csnZPsocgrlb2fYDiWOY=; b=mT0Gj/1hgbfHpr4tBRM0sIv8kGBgibduh6mGLVFoszyYauSXlp7FNBO4ryODJHkNi0 tYGUuYF+EMS0BzEXPKp9l3Fs/9jRiab58j3H835hiKh9MUdsZOsifalrpQyREpY9Gx/T Wdkj5JipzbWXHC9NfXrckH7J0sYhgXKE+hv+iKh7i1eEMS0pzxT88WK7vLhp9uaUM2Ux XlhpEk4EcjhdFDnb2y7HU/8SEfux2krNwImfa7XQxVT6kGThwswMniIPrPAm2RUxahmP 7Lcv5e8AGDmG+ZMsYUk7zn5U7eLy2wfB1e55QQKxCYzfkCwklPXpS24pfhxcE0TKElBx Iv/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YeygMNC1bvvP4dcp+sNmGP0csnZPsocgrlb2fYDiWOY=; b=4ZR/QGSh9D49oC9CrHQtSNx1BpBsgHqHPr5O1If196vrrUr9ENYUgV9WJ8ZK3OeVHC QWh3c1uT1mViaQ6/GS2IUbmoyulGstyOQBrP1ysffSP+se1ZsaLBZe/RJkueaSgCwIlJ 0OvR+cuaVDkHX2UzchnlTZgeiBLFgwpimn/iwCkfoM5e+n1MOYuuXDjYcka5e9x8puDo qLXXMz53Wtzc1GYNyyRLxmCsildr5rg5XjXAUmZeBt3UbZPWziqjDmfiMt5+6BVlXzsc gsCA0pgXQf2wbzUiXGUx/MlIj02BoPaMWb8NCad95//Ti5HxEqLHIxAdsbO3Mi2HBe+K deJA== X-Gm-Message-State: ACrzQf3de6RkEeVJh3WvkZ436g0wkRpD7vcfCIM6eI2l5DU/visvRxlT QeR36NhzDwDhAK/H4n4VmvNAYQm5hPKzzfWdew4r8ZJJ X-Google-Smtp-Source: AMsMyM4i0/+LPtnjst3vkT/fbWzXVFqE9ysDPhBIrR5lM0MKIcPUs4V6vo+39FROICRRSsLKk5L03kciNgz3lD91nkM= X-Received: by 2002:a17:90b:4a09:b0:20d:5b67:1496 with SMTP id kk9-20020a17090b4a0900b0020d5b671496mr6175529pjb.67.1666817503002; Wed, 26 Oct 2022 13:51:43 -0700 (PDT) MIME-Version: 1.0 References: <20221025170519.314511-1-hannes@cmpxchg.org> In-Reply-To: From: Yang Shi Date: Wed, 26 Oct 2022 13:51:22 -0700 Message-ID: Subject: Re: [PATCH] mm: vmscan: split khugepaged stats from direct reclaim stats To: Johannes Weiner Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Eric Bergen Content-Type: text/plain; charset="UTF-8" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666817504; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YeygMNC1bvvP4dcp+sNmGP0csnZPsocgrlb2fYDiWOY=; b=YQDU03ncO1mD5L5phKjyKdKt3r6E3hQgbBhWb9f07wZ5M8fky0yWQfN4gAoMUwekyK0TGY 711wgEBD0xunYOVJJY1NLmK/ws6kF7ec2IiQNCrNs71bN/7E0/RQA0H3ltlAZ7slv7HYm2 4hP3IGN9Cfsu7O8jjbWr/hlUSLsrwQ0= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="mT0Gj/1h"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=shy828301@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666817504; a=rsa-sha256; cv=none; b=WZgxQ/+NaIpHWXG8D4Jd+JKG3WBmee2QFxJUFvujRvmt/riiF/mznh9qMAGUiJAfY0wD9w vQ2FU8eTm3jjX83VlWPcEG9lhSNgWV/TBcasBmfaxqr5TECWI2pSMYPwqSB0n9l6+LIkh6 JhQJuDSlLtZOVdVUCFQc5OGYAXA6yzQ= X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 2512F40008 X-Rspam-User: Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="mT0Gj/1h"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Stat-Signature: yrtaodr5g17pyt9cfy7thka6awy5jxro X-HE-Tag: 1666817503-43026 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Oct 26, 2022 at 10:32 AM Johannes Weiner wrote: > > On Tue, Oct 25, 2022 at 02:53:01PM -0700, Yang Shi wrote: > > On Tue, Oct 25, 2022 at 1:54 PM Johannes Weiner wrote: > > > > > > On Tue, Oct 25, 2022 at 12:40:15PM -0700, Yang Shi wrote: > > > > On Tue, Oct 25, 2022 at 10:05 AM Johannes Weiner wrote: > > > > > > > > > > Direct reclaim stats are useful for identifying a potential source for > > > > > application latency, as well as spotting issues with kswapd. However, > > > > > khugepaged currently distorts the picture: as a kernel thread it > > > > > doesn't impose allocation latencies on userspace, and it explicitly > > > > > opts out of kswapd reclaim. Its activity showing up in the direct > > > > > reclaim stats is misleading. Counting it as kswapd reclaim could also > > > > > cause confusion when trying to understand actual kswapd behavior. > > > > > > > > > > Break out khugepaged from the direct reclaim counters into new > > > > > pgsteal_khugepaged, pgdemote_khugepaged, pgscan_khugepaged counters. > > > > > > > > > > Test with a huge executable (CONFIG_READ_ONLY_THP_FOR_FS): > > > > > > > > > > pgsteal_kswapd 1342185 > > > > > pgsteal_direct 0 > > > > > pgsteal_khugepaged 3623 > > > > > pgscan_kswapd 1345025 > > > > > pgscan_direct 0 > > > > > pgscan_khugepaged 3623 > > > > > > > > There are other kernel threads or works may allocate memory then > > > > trigger memory reclaim, there may be similar problems for them and > > > > someone may try to add a new stat. So how's about we make the stats > > > > more general, for example, call it "pg{steal|scan}_kthread"? > > > > > > I'm not convinved that's a good idea. > > > > > > Can you generally say that userspace isn't indirectly waiting for one > > > of those allocating threads? With khugepaged, we know. > > > > AFAIK, ksm may do slab allocation with __GFP_DIRECT_RECLAIM. > > Right, but ksm also uses __GFP_KSWAPD_RECLAIM. So while userspace > isn't directly waiting for ksm, when ksm enters direct reclaim it's > because kswapd failed. This is of interest to kernel developers. > Userspace will likely see direct reclaim in that scenario as well, so > the ksm direct reclaim counts aren't liable to confuse users. > > Khugepaged on the other hand will *always* reclaim directly, even if > there is no memory pressure or kswapd failure. The direct reclaim > counts there are misleading to both developers and users. > > What it really should be is pgscan_nokswapd_nouserprocesswaiting, but > that just seems kind of long ;-) > > I'm also not sure anybody but khugepaged is doing direct reclaim > without kswapd reclaim. It seems unlikely we'll get more of those. IIUC you actually don't care about how many direct reclaim are triggered by khugepaged, but you would like to separate the direct reclaim stats between that are triggered directly by userspace actions, which may stall userspace, and that aren't, which don't stall userspace. If so it doesn't sound that important to distinguish whether the direct reclaim are triggered by khugepaged or other kernel threads even though other kthreads are not liable to confuse users IMHO. > > > Some device mapper drivers may do heavy lift in the work queue, for > > example, dm-crypt, particularly for writing. > > Userspace will wait for those through dirty throttling. We'd want to Not guaranteed. Anyway I just thought pgscan_khugepaged might be more confusing for the users even the developers who are not familiar with THP/khugepaged. > know about kswapd failures in that case - again, without them being > muddied by khugepaged. > > > > And those other allocations are usually ___GFP_KSWAPD_RECLAIM, so if > > > they do direct reclaim, we'd probably want to know that kswapd is > > > failing to keep up (doubly so if userspace is waiting). In a shared > > > kthread counter, khugepaged would again muddy the waters.