From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18CF7C433FE for ; Wed, 26 Oct 2022 17:32:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE28F8E0003; Wed, 26 Oct 2022 13:32:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A92358E0001; Wed, 26 Oct 2022 13:32:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95A6C8E0003; Wed, 26 Oct 2022 13:32:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 854FD8E0001 for ; Wed, 26 Oct 2022 13:32:23 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 512E4120693 for ; Wed, 26 Oct 2022 17:32:23 +0000 (UTC) X-FDA: 80063794566.04.D3CCE3F Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf09.hostedemail.com (Postfix) with ESMTP id 4760414003E for ; Wed, 26 Oct 2022 17:32:21 +0000 (UTC) Received: by mail-qv1-f46.google.com with SMTP id x15so12142139qvp.1 for ; Wed, 26 Oct 2022 10:32:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=RjKfksPlBhlwYLzBgcKzpziSVwI69KcAuIfLwh+ulWo=; b=QxzeJ/8gAFock46G6AynUz67ur9KujA23UFuhm9X8nKa5ZTKZpdCTPiZaGIjZh6/iQ CUgWzthusrsB7Ny/aynnKAhUgWn3XoloiLODTi/PLWQSwuq3B/LlpXIJuhfHqcE/1H4H Nm8L2FtRBj1/ZFFRVE+4dxftrnSsmcMKGAKbNst3KexHj2hKHpBTQRXD7agBjLXMVTZv DLg8sS3HSUYvMc8Q2pPGmAgWgs0gEZ8++VBmd5KWG5qi9VLDrNmAuHvyH6QIj3EtMG0L oDdCVpJO0Mj0NhqOQ6wZc2lzO/7O5F3t4yu26Lfyi8b30AriWwjXVAcHNv/frQs8Q3rG n5dQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=RjKfksPlBhlwYLzBgcKzpziSVwI69KcAuIfLwh+ulWo=; b=dZMEnh3Fp42GM+NMfdP/1FT+jb6yMBfMJIbJZqMpYpsAbJ/M1xHzSwIXlvNbJ5lZWo 8YidXl97T6W7MmA/khXd07LZ3eaKJNmCCZuxB5c6DRShECSASZ2bG7tZyAQrkNjwgvM3 CF95vqnGj1HKm3u0/FmpaRDShKI898cDrRSE3S+06oPldWxh3cGCNjN3P+mR9RXPvV4A PHRDpOlOjGk6MpkEFPlwupYSzz0iR9VOjhLDZAYisaaV1xaxO7E8Ruo7ixH/kQ5Ku7S3 i4ygoe3N7YfL/1co5Ljudze7JbDbr7MZuX6zfMPaQbGWrG619r/beVn83vGgqaORtQAD efog== X-Gm-Message-State: ACrzQf0oEf9uYSrMICcMk+uuVLAcVr2EvlA9mx7Gvtey2FXDVbdI4VK9 g1HefLvV2KIfaDpd1TpelDhrDQ== X-Google-Smtp-Source: AMsMyM4zC2bor82A6TRyJteIhhSCalmnmm5YkOvC3A7NuQKtQ6+exeNR7sYQ5Pr1gVPPygNaGtNTqg== X-Received: by 2002:a05:6214:ac7:b0:4bb:61e7:18eb with SMTP id g7-20020a0562140ac700b004bb61e718ebmr3576201qvi.55.1666805540383; Wed, 26 Oct 2022 10:32:20 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::25f1]) by smtp.gmail.com with ESMTPSA id x11-20020a05620a258b00b006ceb933a9fesm4351579qko.81.2022.10.26.10.32.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Oct 2022 10:32:20 -0700 (PDT) Date: Wed, 26 Oct 2022 13:32:20 -0400 From: Johannes Weiner To: Yang Shi Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Eric Bergen Subject: Re: [PATCH] mm: vmscan: split khugepaged stats from direct reclaim stats Message-ID: References: <20221025170519.314511-1-hannes@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666805541; a=rsa-sha256; cv=none; b=0hZRZMv7j1n+YRCunUAqkn05Hn88xohzsR5TdGLSZmiYIcyzEttZtcZn/kcr6+w135sgLW U+j23gM9ec5lW0kk0EiMVj+pXXi+jkerGa4545zzpIPJV4pmemP+UInG/GdQg+osmANFRE QdnBeJ7npDmkM2UgL8eO1++iI3y4cGE= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b="QxzeJ/8g"; spf=pass (imf09.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666805541; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RjKfksPlBhlwYLzBgcKzpziSVwI69KcAuIfLwh+ulWo=; b=H8XHfCzk2pfEet6c1nlORcUkfFelgE6/1cnA93nxvXP+MfhVIH0leVyS8z8MaEqsTxOes7 3lY2fiJ5apItRDtnoWqIBssqlDwlNSg04cb0uQcwAKdNEd9boAui1XgrZv4Nxr7QByytLn /TUfQTSlbQfa/uQ1N9ZpUAUp3svqKTg= X-Rspam-User: X-Rspamd-Queue-Id: 4760414003E Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b="QxzeJ/8g"; spf=pass (imf09.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.46 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Stat-Signature: bh5mi5pkwt71phbid3rewbn5xniiii7a X-Rspamd-Server: rspam03 X-HE-Tag: 1666805541-267593 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Oct 25, 2022 at 02:53:01PM -0700, Yang Shi wrote: > On Tue, Oct 25, 2022 at 1:54 PM Johannes Weiner wrote: > > > > On Tue, Oct 25, 2022 at 12:40:15PM -0700, Yang Shi wrote: > > > On Tue, Oct 25, 2022 at 10:05 AM Johannes Weiner wrote: > > > > > > > > Direct reclaim stats are useful for identifying a potential source for > > > > application latency, as well as spotting issues with kswapd. However, > > > > khugepaged currently distorts the picture: as a kernel thread it > > > > doesn't impose allocation latencies on userspace, and it explicitly > > > > opts out of kswapd reclaim. Its activity showing up in the direct > > > > reclaim stats is misleading. Counting it as kswapd reclaim could also > > > > cause confusion when trying to understand actual kswapd behavior. > > > > > > > > Break out khugepaged from the direct reclaim counters into new > > > > pgsteal_khugepaged, pgdemote_khugepaged, pgscan_khugepaged counters. > > > > > > > > Test with a huge executable (CONFIG_READ_ONLY_THP_FOR_FS): > > > > > > > > pgsteal_kswapd 1342185 > > > > pgsteal_direct 0 > > > > pgsteal_khugepaged 3623 > > > > pgscan_kswapd 1345025 > > > > pgscan_direct 0 > > > > pgscan_khugepaged 3623 > > > > > > There are other kernel threads or works may allocate memory then > > > trigger memory reclaim, there may be similar problems for them and > > > someone may try to add a new stat. So how's about we make the stats > > > more general, for example, call it "pg{steal|scan}_kthread"? > > > > I'm not convinved that's a good idea. > > > > Can you generally say that userspace isn't indirectly waiting for one > > of those allocating threads? With khugepaged, we know. > > AFAIK, ksm may do slab allocation with __GFP_DIRECT_RECLAIM. Right, but ksm also uses __GFP_KSWAPD_RECLAIM. So while userspace isn't directly waiting for ksm, when ksm enters direct reclaim it's because kswapd failed. This is of interest to kernel developers. Userspace will likely see direct reclaim in that scenario as well, so the ksm direct reclaim counts aren't liable to confuse users. Khugepaged on the other hand will *always* reclaim directly, even if there is no memory pressure or kswapd failure. The direct reclaim counts there are misleading to both developers and users. What it really should be is pgscan_nokswapd_nouserprocesswaiting, but that just seems kind of long ;-) I'm also not sure anybody but khugepaged is doing direct reclaim without kswapd reclaim. It seems unlikely we'll get more of those. > Some device mapper drivers may do heavy lift in the work queue, for > example, dm-crypt, particularly for writing. Userspace will wait for those through dirty throttling. We'd want to know about kswapd failures in that case - again, without them being muddied by khugepaged. > > And those other allocations are usually ___GFP_KSWAPD_RECLAIM, so if > > they do direct reclaim, we'd probably want to know that kswapd is > > failing to keep up (doubly so if userspace is waiting). In a shared > > kthread counter, khugepaged would again muddy the waters.