From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 455E3C43217 for ; Thu, 24 Feb 2022 16:58:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 630458D0019; Thu, 24 Feb 2022 11:58:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DFEB8D0011; Thu, 24 Feb 2022 11:58:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29D218D0018; Thu, 24 Feb 2022 11:58:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id DE4E78D0018 for ; Thu, 24 Feb 2022 11:58:43 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B285A218DF for ; Thu, 24 Feb 2022 16:58:43 +0000 (UTC) X-FDA: 79178282526.01.E0A3C86 Received: from mail-ot1-f74.google.com (mail-ot1-f74.google.com [209.85.210.74]) by imf04.hostedemail.com (Postfix) with ESMTP id 162454000B for ; Thu, 24 Feb 2022 16:58:42 +0000 (UTC) Received: by mail-ot1-f74.google.com with SMTP id d3-20020a9d4f03000000b005af2a6014f7so1796814otl.23 for ; Thu, 24 Feb 2022 08:58:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=2qICqefspcT3l9C7ksTBXbsPwv28Z6xEUTC2hsoiyJ0=; b=UoVRBBqx5GxP6ot4XbI8IBaC35V3O+MR3N1hvAtrfsXfDrYp+BGmznJW288ysWd/Sc BtSB+KyY4u2q2grJZHNOPVanv4BNtJiXzRBKP8skmt4NJaTy75IQlzRkhduQ90HdWjaQ 5LoeqfWlPcjsjpQ220pPcq+Bn787mIYCyrPdrLyCsH1/ggbm9C1uCvb3v4c82b4M/uIY t65xqIdgB24kB24ieKLf04C7IrAeEGknJGnximmRCgGLSsayKP3tyyhzwrZo2qZdthHw JtsB7EQt7BCOINbu4Ih6j+cBQi2ehwd+WJ7CVaIESPmXaVrM2mzynerDHxbcSPbCkqbM +DQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=2qICqefspcT3l9C7ksTBXbsPwv28Z6xEUTC2hsoiyJ0=; b=uGTy8DAbKt7SxfGLM5B59inDHZgtxaqCkj0aZmYVqcD1WFQu6tv7x587KWD6wFsYhb RSX6dJoUwBN3NdLimybkuJ0zFig9nNyH1Z0lHrTzkAJLneivLhQYpE2dieNV466qIgQ2 wzYb+OspUK5FYePGiJmhdm4bZaMxWI9ge2qbZ14PLhkl/uemXOorb5qHdGuumYns6U59 W4h8TdSVcZytol7UDydb542/XqspN72Li9zUv3ALbv72bSn2ZGL7VH+9EsADLFdoZQZo unI1bPUMjdk8JbVQX2Z33Nt1VSsQWyZ/Ox5K65CZDpA6MjCTW2qgygVMKUG+j05eBdVx DLpQ== X-Gm-Message-State: AOAM5329D7zdZq17wAcfe5ZRisojVgVf/StBInWprfmovj8sHYpQ+Fwh oWyAlwMAfbCUPiJc5IrNPh4SqM4R5+iTyA== X-Google-Smtp-Source: ABdhPJyFIphMPxJ0gKfEKeiOXX3q3oozqt2Tqy0JvbeWBFYPy16bEnbA8325Wls8bJsAczMpGcar6iEd9VYy4A== X-Received: from shakeelb.svl.corp.google.com ([2620:15c:2cd:202:8c61:13e8:87c2:e5f0]) (user=shakeelb job=sendgmr) by 2002:a05:6808:1a0d:b0:2d5:3f8c:9132 with SMTP id bk13-20020a0568081a0d00b002d53f8c9132mr1833836oib.51.1645721922208; Thu, 24 Feb 2022 08:58:42 -0800 (PST) Date: Thu, 24 Feb 2022 08:58:38 -0800 In-Reply-To: Message-Id: <20220224165838.oir5clpkkqpstpx3@google.com> Mime-Version: 1.0 References: Subject: Re: Regression in workingset_refault latency on 5.15 From: Shakeel Butt To: Daniel Dao Cc: Ivan Babrou , kernel-team , Linux MM , Johannes Weiner , Roman Gushchin , Feng Tang , Michal Hocko , Hillf Danton , "Michal =?utf-8?Q?Koutn=C3=BD?=" , Andrew Morton , Linus Torvalds Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 162454000B X-Stat-Signature: 8ggyjk7ds6pd4hu983jmg4x9c1ccwn5r Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=UoVRBBqx; spf=pass (imf04.hostedemail.com: domain of 3QrkXYggKCCsZOHRLLSINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--shakeelb.bounces.google.com designates 209.85.210.74 as permitted sender) smtp.mailfrom=3QrkXYggKCCsZOHRLLSINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-HE-Tag: 1645721922-164950 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000015, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Feb 24, 2022 at 02:46:27PM +0000, Daniel Dao wrote: [...] > 3) Summary of stack traces when mem_cgroup_flush_stats is over 5ms Can you please check if flush_memcg_stats_dwork() appears in any stack traces at all? [...] > > > > If that patch did not help then we either have to optimize rstat > > flushing or further increase the update buffer which is nr_cpus * 32. > I think overall, we don't see any significant improvements. The > appearance of > expensive mem_cgroup_flush_stats is still there. Thanks for testing. At the moment I am suspecting the async worker is not getting the CPU. Can you share your CONFIG_HZ setting? Also can you try the following patch and see if that helps otherwise keep halving the delay (i.e. 2HZ -> HZ -> HZ/2 -> ...) and find at what value the issue you are seeing get resolved? diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 32ba963ebf2e..198a7baaeeb7 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -685,7 +685,7 @@ void mem_cgroup_flush_stats(void) static void flush_memcg_stats_dwork(struct work_struct *w) { __mem_cgroup_flush_stats(); - queue_delayed_work(system_unbound_wq, &stats_flush_dwork, 2UL*HZ); + queue_delayed_work(system_unbound_wq, &stats_flush_dwork, HZ); } /**