From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B66C4ECAAA1 for ; Thu, 27 Oct 2022 14:15:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6F258E0002; Thu, 27 Oct 2022 10:15:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DF6A88E0001; Thu, 27 Oct 2022 10:15:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C70058E0002; Thu, 27 Oct 2022 10:15:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B18BC8E0001 for ; Thu, 27 Oct 2022 10:15:02 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 52FE61610CB for ; Thu, 27 Oct 2022 14:15:02 +0000 (UTC) X-FDA: 80066926044.22.AFDFCB8 Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by imf22.hostedemail.com (Postfix) with ESMTP id 9E4D2C000F for ; Thu, 27 Oct 2022 14:15:00 +0000 (UTC) Received: by mail-qt1-f180.google.com with SMTP id z6so1183337qtv.5 for ; Thu, 27 Oct 2022 07:15:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=FiB4/nl2jh2fpoxzZJWIvh2hpwEiUdFi4UfLtVCENQg=; b=AeTtgS/zZSnwsC9g8GsOldGdwurc9M1oBkLYH2kp9NS3+l30aKe+SXPdGa0vOpXZqf 66aHfw92eRn6oVxjdSHlIGccu+js8ILxw87BIz48jsc1U4nK8JPCEvzkIdcgv05vNzXl nb8k2vKMXKyvotZm2b6USSq996c0lHEI/ED1DXLKOnsyr508MRXK1znMs9l4tHwVuk1f xlIRfyJswp8J/5Xjv7SGaMkB47zRA1nIku4pmZl1sWdRXOG41P7rtA8cohc4PL2t4fks h83PvU+W14WKzD1GpFM1mXaKKde4rATCcbHACvpZPzuftI6Q8p6uAS6nPg3AiR1dl0wp +HjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=FiB4/nl2jh2fpoxzZJWIvh2hpwEiUdFi4UfLtVCENQg=; b=biWGVqJzpqHAj+DPT1t6r8hVO8EkDBh5Q/pZ9f7+qFse44kovG9WhwHhtypa/AKwMB tA3zxZkfQQLM05GgnX00nsGIiIWlg6hL6z5JE9hIbJuFgejyH8Mr8E40f74SPfdGkZ0D dI4fLlx5wGfie0vDhnvPpdT5cfoIzOuiXkPR0nCX6MOP5MW3iMhNWhJoToTfZQhSXMIL fAB89lP2LoN/70cDGKaLCck+ocPvX5DuWND3RYKsfFN4Vvz0NUfotEyYP90zuAw/tvE/ 7mmGQykAn4GV+j5mOrOWi2aL9gZkvxuaceecrLBu5AlrR5koHGrD32vfJlHf+2Em3Ze+ EPOg== X-Gm-Message-State: ACrzQf3dB2Q/ANS9dAXDQMlP7+iU14xdJsisF8C1D+tx1yiTQyqzgTHf 6EwPiP2OJ3rlAOZWN6JIdKdygg== X-Google-Smtp-Source: AMsMyM6LcCaZZAUoNCcj7eWRFmN1YMhnldHivBKR+dETtiflk/Op4/eS2tIEbNoo6ZeiRY8Vs5dytA== X-Received: by 2002:ac8:5707:0:b0:39d:c40:cf51 with SMTP id 7-20020ac85707000000b0039d0c40cf51mr32277314qtw.102.1666880099687; Thu, 27 Oct 2022 07:14:59 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::25f1]) by smtp.gmail.com with ESMTPSA id r6-20020ac85c86000000b003431446588fsm966231qta.5.2022.10.27.07.14.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Oct 2022 07:14:59 -0700 (PDT) Date: Thu, 27 Oct 2022 10:15:00 -0400 From: Johannes Weiner To: Yosry Ahmed Cc: Yang Shi , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Eric Bergen Subject: Re: [PATCH] mm: vmscan: split khugepaged stats from direct reclaim stats Message-ID: References: <20221025170519.314511-1-hannes@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666880101; a=rsa-sha256; cv=none; b=RzVS//UuVfeZOO7WHdQ/vOtJhN8mlZyaBKd4gpbCctbbYn1/+4e/DoiuQ29ig+ePv3SF+x WXQ18DcXQZOHirmJ2H0eele45oZmoetr0md1AyLgVMr3xC7SFBuP8majtTr6AK+/dMdc22 HqxxIZE9fAg3VwDIY9PClm81LmIyjwU= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b="AeTtgS/z"; spf=pass (imf22.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.180 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666880101; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FiB4/nl2jh2fpoxzZJWIvh2hpwEiUdFi4UfLtVCENQg=; b=VsrSxt1/FR+6ydnmMNVNqg0tsr2z2uDKNLxZCfLgZCniKHBButy91PO2HfcOwGrRxbr33D /YDJ0ZCHsVXvqd0Cos1LNBHggkBN+OP+Oa4wlgVyV2V6mZFeFufqHxorFGie1M2GB0xbjX dBc4jGjwS4Y/KNKMjy3tSqP5HmKOMFM= X-Rspam-User: X-Rspamd-Queue-Id: 9E4D2C000F Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b="AeTtgS/z"; spf=pass (imf22.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.180 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Stat-Signature: 3axdzsph6fr7kgtc6r9igzmobpueap8k X-Rspamd-Server: rspam03 X-HE-Tag: 1666880100-99005 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Oct 26, 2022 at 07:41:21PM -0700, Yosry Ahmed wrote: > On Wed, Oct 26, 2022 at 1:51 PM Yang Shi wrote: > > > > On Wed, Oct 26, 2022 at 10:32 AM Johannes Weiner wrote: > > > > > > On Tue, Oct 25, 2022 at 02:53:01PM -0700, Yang Shi wrote: > > > > On Tue, Oct 25, 2022 at 1:54 PM Johannes Weiner wrote: > > > > > > > > > > On Tue, Oct 25, 2022 at 12:40:15PM -0700, Yang Shi wrote: > > > > > > On Tue, Oct 25, 2022 at 10:05 AM Johannes Weiner wrote: > > > > > > > > > > > > > > Direct reclaim stats are useful for identifying a potential source for > > > > > > > application latency, as well as spotting issues with kswapd. However, > > > > > > > khugepaged currently distorts the picture: as a kernel thread it > > > > > > > doesn't impose allocation latencies on userspace, and it explicitly > > > > > > > opts out of kswapd reclaim. Its activity showing up in the direct > > > > > > > reclaim stats is misleading. Counting it as kswapd reclaim could also > > > > > > > cause confusion when trying to understand actual kswapd behavior. > > > > > > > > > > > > > > Break out khugepaged from the direct reclaim counters into new > > > > > > > pgsteal_khugepaged, pgdemote_khugepaged, pgscan_khugepaged counters. > > > > > > > > > > > > > > Test with a huge executable (CONFIG_READ_ONLY_THP_FOR_FS): > > > > > > > > > > > > > > pgsteal_kswapd 1342185 > > > > > > > pgsteal_direct 0 > > > > > > > pgsteal_khugepaged 3623 > > > > > > > pgscan_kswapd 1345025 > > > > > > > pgscan_direct 0 > > > > > > > pgscan_khugepaged 3623 > > > > > > > > > > > > There are other kernel threads or works may allocate memory then > > > > > > trigger memory reclaim, there may be similar problems for them and > > > > > > someone may try to add a new stat. So how's about we make the stats > > > > > > more general, for example, call it "pg{steal|scan}_kthread"? > > > > > > > > > > I'm not convinved that's a good idea. > > > > > > > > > > Can you generally say that userspace isn't indirectly waiting for one > > > > > of those allocating threads? With khugepaged, we know. > > > > > > > > AFAIK, ksm may do slab allocation with __GFP_DIRECT_RECLAIM. > > > > > > Right, but ksm also uses __GFP_KSWAPD_RECLAIM. So while userspace > > > isn't directly waiting for ksm, when ksm enters direct reclaim it's > > > because kswapd failed. This is of interest to kernel developers. > > > Userspace will likely see direct reclaim in that scenario as well, so > > > the ksm direct reclaim counts aren't liable to confuse users. > > > > > > Khugepaged on the other hand will *always* reclaim directly, even if > > > there is no memory pressure or kswapd failure. The direct reclaim > > > counts there are misleading to both developers and users. > > > > > > What it really should be is pgscan_nokswapd_nouserprocesswaiting, but > > > that just seems kind of long ;-) > > > > > > I'm also not sure anybody but khugepaged is doing direct reclaim > > > without kswapd reclaim. It seems unlikely we'll get more of those. > > > > IIUC you actually don't care about how many direct reclaim are > > triggered by khugepaged, but you would like to separate the direct > > reclaim stats between that are triggered directly by userspace > > actions, which may stall userspace, and that aren't, which don't stall > > userspace. If so it doesn't sound that important to distinguish > > whether the direct reclaim are triggered by khugepaged or other kernel > > threads even though other kthreads are not liable to confuse users > > IMHO. I feel like I've sufficiently explained my reason for wanting to separate out the __GFP_KSWAPD_RECLAIM special case from other sites. > My 2c, if we care about direct reclaim as in reclaim that may stall > user space application allocations, then there are other reclaim > contexts that may pollute the direct reclaim stats. For instance, > proactive reclaim, or reclaim done by writing a limit lower than the > current usage to memory.max or memory.high, as they are not done in > the context of the application allocating memory. > > At Google, we have some internal direct reclaim memcg statistics, and > the way we handle this is by passing a flag from such contexts to > try_to_free_mem_cgroup_pages() in the reclaim_options arg. This flag > is echod into a scan_struct bit, which we then use to filter out > direct reclaim operations that actually cause latencies in user space > allocations. > > Perhaps something similar might be more generic here? I am not sure > what context khugepaged reclaims memory from, but I think it's not a > memcg context, so maybe we want to generalize the reclaim_options arg > to try_to_free_pages() or whatever interface khugepaged uses to free > memory. So at the /proc/vmstat level, I'm not sure it matters much because it doesn't count any cgroup_reclaim() activity. But at the cgroup level, it sure would be nice to split out proactive reclaim churn. Both in terms of not polluting direct reclaim counts, but also for *knowing* how much proactive reclaim is doing. Do you have separate counters for this?