From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F23CC433EF for ; Thu, 24 Feb 2022 18:01:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B07F58D0002; Thu, 24 Feb 2022 13:01:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB79E8D0001; Thu, 24 Feb 2022 13:01:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A7628D0002; Thu, 24 Feb 2022 13:01:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 877D48D0001 for ; Thu, 24 Feb 2022 13:01:10 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5E83A16FD for ; Thu, 24 Feb 2022 18:01:10 +0000 (UTC) X-FDA: 79178439900.14.3F5EB6B Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf04.hostedemail.com (Postfix) with ESMTP id DAF0E40043 for ; Thu, 24 Feb 2022 18:01:08 +0000 (UTC) Received: by mail-pg1-f169.google.com with SMTP id e6so989763pgn.2 for ; Thu, 24 Feb 2022 10:01:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zTmqeICKCOBXWeUdflnc+d2/KF2tNcNryjV/9zqqr9Y=; b=VqKqrqqlyNSLkBs1NWJ8WU27KMZh4d6EtRyZK3iorxEDfDxSrqvB3SeeQEtAhflIAa bbYlihnfnrlrAbDKtJJMAtAJ6aERLamja1i5XxqYBBNhs8dJTQrpSOCg72qSZUCuy8pW FWQvKb7D8coJYhezvYUTwfMvWCht5VZMufayaWrD540skdHk9ut5oOkvlbD1wTXpx/kV RTYK8ew7vxi+6ybG1107dBUpFdrMavbXrY+rJ9yld2Tk9s3e8rW2eec2xFuNRjdkjJei BhdTfXSRUAErU32b2dXag8zsg1ABbv2awVDcYSPsHz+MMA9Bw5E4GcOSUfdzT6V43i3T MshQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zTmqeICKCOBXWeUdflnc+d2/KF2tNcNryjV/9zqqr9Y=; b=v6IMH8WI+rvx83d4JUsFl4dvrc3xBpRSFE4wFWsandPb+tVYRcDxG8jzhjNT24/Udp Zthfffej136Oup0Ut761XfI7GUgbW8iGs+eYe0XCcR4/i+ztuqmFVFc3GmuKQIE8F92n r2lZtUpGvLXzHB6fyrHYR/89MyHwUKK57RK6XK0TSrzJlH4bH8EI/37VbJOYGmicPvk+ 3/cgVv2pdR6Q5K+liyfOgw0/gC28sNajpY9IAR7wM061kfU/Y8tfHkE41iGRAq5B6jiK vJltrg3MqvBL89UAjhMstbKtz4nlDYIQ9quEM7Dwk6XPWk8sp6TDZ1qbquLppMR7kpWW iO1Q== X-Gm-Message-State: AOAM531FwyKR8eqahLAqje6+GtAPtHkYuMBrNcXpTT08c2MrPrHHk1Qz c8cIuGLPKeobjGctFdbmM1LXfH8yJXYUNVWpxt16uw== X-Google-Smtp-Source: ABdhPJxjY+Hgay1trUCUpS6xcO5JDD72J1drLbk2ucpbenH6BONhrsG0kYK9Rk1vd30Z2XthBqg1o4DQ6EPfZdFoLBE= X-Received: by 2002:a05:6a00:21c8:b0:4e1:69f6:1c23 with SMTP id t8-20020a056a0021c800b004e169f61c23mr3957645pfj.8.1645725667501; Thu, 24 Feb 2022 10:01:07 -0800 (PST) MIME-Version: 1.0 References: <20220224165838.oir5clpkkqpstpx3@google.com> In-Reply-To: From: Shakeel Butt Date: Thu, 24 Feb 2022 10:00:55 -0800 Message-ID: Subject: Re: Regression in workingset_refault latency on 5.15 To: Daniel Dao Cc: Ivan Babrou , kernel-team , Linux MM , Johannes Weiner , Roman Gushchin , Feng Tang , Michal Hocko , Hillf Danton , =?UTF-8?Q?Michal_Koutn=C3=BD?= , Andrew Morton , Linus Torvalds Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: DAF0E40043 X-Stat-Signature: yqhsk7x9bsztfq88fhfa4w4ognif7zeb Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VqKqrqql; spf=pass (imf04.hostedemail.com: domain of shakeelb@google.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1645725668-30442 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Feb 24, 2022 at 9:34 AM Daniel Dao wrote: > [...] > > Thanks for testing. At the moment I am suspecting the async worker is > > not getting the CPU. Can you share your CONFIG_HZ setting? Also can you > > try the following patch and see if that helps otherwise keep halving the > > delay (i.e. 2HZ -> HZ -> HZ/2 -> ...) and find at what value the issue > > you are seeing get resolved? > > We have CONFIG_HZ=1000. We can try to increase the frequency of async flush, but > that seems like a not great bandaid. Is it possible to remove > mem_cgroup_flush_stats() > from workingset_refault, or at least scope it down to some targeted cgroup so > we don't need to flush from root with potentially large sets of > cgroups to walk ? I actually wanted to know what would be the good frequency of rstat flushing for your workload. I am not planning to propose to change the default frequency. Anyways I am thinking of introducing mem_cgroup_flush_stats_asyn() which will schedule flush_memcg_stats_dwork() without delay. Let me prepare the patch based on 5.15-stable for you to test.