From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6320C7EE23 for ; Wed, 24 May 2023 12:34:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A49A6B0074; Wed, 24 May 2023 08:34:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6051A6B0075; Wed, 24 May 2023 08:34:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4CE2F900002; Wed, 24 May 2023 08:34:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3EBEF6B0074 for ; Wed, 24 May 2023 08:34:43 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0B8061C72C2 for ; Wed, 24 May 2023 12:34:43 +0000 (UTC) X-FDA: 80825092446.30.C18EE6C Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf11.hostedemail.com (Postfix) with ESMTP id 117024000C for ; Wed, 24 May 2023 12:34:40 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=eDp4Qudo; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf11.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684931681; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AiiMIvYGdbIRGL9zQDfeMCY6Hk4lfNGeYEXmOzeLdh0=; b=NV1itb9fbVGiUvOe1F25Xy0MhAwMxlZdipi2DnvpsZXhyTvfZW0Q7dM997i3GovmnmnfjU Y4M0QXXvYd3xvjpW8fOG3mdpinHFLSxfBnE9LxNeaX4d6UQuuVh8mufz6ddMEdM3U0CNvG GfnNRgnC4YvVqa4VhT1bVp3kxx8Vg2s= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=eDp4Qudo; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf11.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684931681; a=rsa-sha256; cv=none; b=PuTJDlhicgtGwNxnGSlGbTeCOPffXwdX+aZOJP4LxBb9nkipssO+N4Sj7pbMtlLbRPpOFW E/upYbNNSpT1q5cbdeMJjuQKNsuvkoPuoZx2Ow+oLPL/+1kBmlJwEIBFqIpcuyFrfz1riL MgWIapPGA44B1FDcIR/eKoAGjU8kHKg= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 6AD231F8BF; Wed, 24 May 2023 12:34:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1684931679; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=AiiMIvYGdbIRGL9zQDfeMCY6Hk4lfNGeYEXmOzeLdh0=; b=eDp4QudooCjdxpuWCFiomINeR63fpz11H/DfOnUwO7iR8RKbLClzPI9G1HSlFw/4jQUlLw U6y+szIJ3lKAPBGQBgvx4Amf9OQVgnABQBWTeI2fWxlZR93ByrtsD7q73sSG+GzkdVa8Sk n09CF444KiotVDN91YA0flskmKbWnIE= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 41FD313425; Wed, 24 May 2023 12:34:39 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id t7NMDV8EbmQeawAAMHmgww (envelope-from ); Wed, 24 May 2023 12:34:39 +0000 Date: Wed, 24 May 2023 14:34:38 +0200 From: Michal Hocko To: Marcelo Tosatti Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka Subject: Re: [PATCH v8 01/13] vmstat: allow_direct_reclaim should use zone_page_state_snapshot Message-ID: References: <20230515180015.016409657@redhat.com> <20230515180138.442505633@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230515180138.442505633@redhat.com> X-Rspam-User: X-Stat-Signature: f6d8iguqxkca8uwge1b3y383afnx4c9y X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 117024000C X-HE-Tag: 1684931680-713929 X-HE-Meta: U2FsdGVkX18zrfg+kjsowMZYUF4a4HsXqQ4LphUMu1OvcQrpZXyjSVfJSJkbJR9ilAZKiqTi8XICHYdpmCT0cM5dMA7CUddLL/FVF6L1XC21Zy9HSqA/QmdVWiiQJg3H67rKF/dRJLIi1B5T9W/EUEsKAhrHdU24ZmJ9uKEC/615Tz2LMvXr7C6wmRi8jAEEMwEc4FWlQIfdKnEJa83k1+tKv0clxZZoD3erKrtSLrZe81IaPEwWmmMlYtw3w+Jf4/ivgsoyhlQ12TxHECBokljGxhx9MWn26a36iftoleXW3KZhrI5dSLoOKgXiY/PRjCN25LrE3qfFIhfuN2qSjmxOgWvlpZ8m8YrUWhIdIvUKSzKH7YhzJU3tOSVi8XwESJ16aCuzn19YPA6X9Boh49k6bAqCz9L/6dOi6pltgOBCsgmxwsNG80wMlNjneEJuwVs3t6JfY/b6gvO0AHgoh9cEHHz8s1PNYPa2HgfcuojN0iQSoLXhpAwbAolzDX7kexxir5zW5DSCl+fLxbbGDRrn5gKKv3O6kNfpNcdx+YcQPdMFHW5QMqL2mr0v+UKvo/HYStxSgNC6vnxb5X32+IKgGxBk8W9+/EPJWDAcecEabxd+z46pE8KZuKUnmOa8MKMN+XHB9JF2y7HGAZRIeyJsapopnYXOh259nyz1TcaPXdLuCXdZRdUlsh2wiYwVU5J8D+x2wL0DLxYFf3DFyZjpS9zeAuwpOutG15yicOrZ5BFsIaev2ekeq0XWQ8s0pOYUet8HSS1Bhe82KjR7x0npE8sYSaMgGpokQnDUrk4UGnzYLDPlQ9aGqlTtHgN0gTEtPE0lZ4DOb0Y5ctJGI0ZNxG1DDoKQy7QWvV9rG0+IRUERfbmoNhkto2tb3jjHTD9Brf9v5RM5iB5jlazrnXmsOUpk3n4hIXRGyleQc1RbLcMcF0MDppL4URnAMwqRydylZDUYaAzWhp46h7/ m/UoSgVJ q5dLnj9Pgh/qvJS0ZnAwFhHCCtcoTf1ElzLophOKdi/KxKG6p7qT0ed9nl0kYEJB5WFgonEYb3N89BLQ38s6myEOfX+/UvS6FAyqZxrkdJIDOcM4tgE/1h6HZHz0RnI1SzzRqSux07oew6x0Ze8fUEEQgdwpcZ1T8fUZ7J0i+9CkyeQmdqE68oOyfJMByQbZGkEYll21ZRaU9PJzdvyviSJHoC3C1RGW2iTEEPNH/ar/OREoBFE374Mldag0yi7/yFC6/KEbarzp+OkQYg6ZFhvXRGQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon 15-05-23 15:00:16, Marcelo Tosatti wrote: > A customer provided evidence indicating that a process > was stalled in direct reclaim: > > - The process was trapped in throttle_direct_reclaim(). > The function wait_event_killable() was called to wait condition > allow_direct_reclaim(pgdat) for current node to be true. > The allow_direct_reclaim(pgdat) examined the number of free pages > on the node by zone_page_state() which just returns value in > zone->vm_stat[NR_FREE_PAGES]. > > - On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. > However, the freelist on this node was not empty. > > - This inconsistent of vmstat value was caused by percpu vmstat on > nohz_full cpus. Every increment/decrement of vmstat is performed > on percpu vmstat counter at first, then pooled diffs are cumulated > to the zone's vmstat counter in timely manner. However, on nohz_full > cpus (in case of this customer's system, 48 of 52 cpus) these pooled > diffs were not cumulated once the cpu had no event on it so that > the cpu started sleeping infinitely. > I checked percpu vmstat and found there were total 69 counts not > cumulated to the zone's vmstat counter yet. > > - In this situation, kswapd did not help the trapped process. > In pgdat_balanced(), zone_wakermark_ok_safe() examined the number > of free pages on the node by zone_page_state_snapshot() which > checks pending counts on percpu vmstat. > Therefore kswapd could know there were 69 free pages correctly. > Since zone->_watermark = {8, 20, 32}, kswapd did not work because > 69 was greater than 32 as high watermark. > > Change allow_direct_reclaim to use zone_page_state_snapshot, which > allows a more precise version of the vmstat counters to be used. > > allow_direct_reclaim will only be called from try_to_free_pages, > which is not a hot path. I have asked this patch to contain a note about testing done for future reference IIRC. You have said that the above scenario is not easy to reproduce and therefore not much of testing has been done but this is a valuable information on its own. > Suggested-by: Michal Hocko > Signed-off-by: Marcelo Tosatti I have also acked the patch. So let me repeat Acked-by: Michal Hocko Btw. the patch should be routed separately as it is a potential fix. > > --- > > Index: linux-vmstat-remote/mm/vmscan.c > =================================================================== > --- linux-vmstat-remote.orig/mm/vmscan.c > +++ linux-vmstat-remote/mm/vmscan.c > @@ -6886,7 +6886,7 @@ static bool allow_direct_reclaim(pg_data > continue; > > pfmemalloc_reserve += min_wmark_pages(zone); > - free_pages += zone_page_state(zone, NR_FREE_PAGES); > + free_pages += zone_page_state_snapshot(zone, NR_FREE_PAGES); > } > > /* If there are no reserves (unexpected config) then do not throttle */ > -- Michal Hocko SUSE Labs