From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37DE9C433E0 for ; Fri, 19 Feb 2021 18:59:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 78E1264EB7 for ; Fri, 19 Feb 2021 18:59:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 78E1264EB7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 696266B0005; Fri, 19 Feb 2021 13:59:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6465A6B006C; Fri, 19 Feb 2021 13:59:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55BE66B006E; Fri, 19 Feb 2021 13:59:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0174.hostedemail.com [216.40.44.174]) by kanga.kvack.org (Postfix) with ESMTP id 3AE896B0005 for ; Fri, 19 Feb 2021 13:59:11 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id CEEF0180B7E0D for ; Fri, 19 Feb 2021 18:59:10 +0000 (UTC) X-FDA: 77835930060.12.166E26D Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf26.hostedemail.com (Postfix) with ESMTP id 67C4E407F8F3 for ; Fri, 19 Feb 2021 18:59:06 +0000 (UTC) IronPort-SDR: JH5f12FlTuWQ7PZUZ4NVPAYlADqRTykdlXuY7qdO0g4zG+k5D+GQmzAZKDdQ/amVbZzTqEJMAo XebUx0lIHH8g== X-IronPort-AV: E=McAfee;i="6000,8403,9900"; a="183146170" X-IronPort-AV: E=Sophos;i="5.81,189,1610438400"; d="scan'208";a="183146170" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Feb 2021 10:59:07 -0800 IronPort-SDR: ydQjHIvRVWQzSbtxEkBhnUQPSo9AYdN4OeN+tifCiaNeCb1Aj2HAAXQSv7DrzF4eB5Hl8v+5wj jAl7jBKIZnaw== X-IronPort-AV: E=Sophos;i="5.81,189,1610438400"; d="scan'208";a="378989329" Received: from schen9-mobl.amr.corp.intel.com ([10.251.10.112]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Feb 2021 10:59:07 -0800 Subject: Re: [PATCH v2 2/3] mm: Force update of mem cgroup soft limit tree on usage excess To: Michal Hocko Cc: Andrew Morton , Johannes Weiner , Vladimir Davydov , Dave Hansen , Ying Huang , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org References: <06f1f92f1f7d4e57c4e20c97f435252c16c60a27.1613584277.git.tim.c.chen@linux.intel.com> From: Tim Chen Message-ID: <884d7559-e118-3773-351d-84c02642ca96@linux.intel.com> Date: Fri, 19 Feb 2021 10:59:05 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Stat-Signature: rzizdfwo7ypxt83txr1uo8k8gup9tma4 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 67C4E407F8F3 Received-SPF: none (linux.intel.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=mga14.intel.com; client-ip=192.55.52.115 X-HE-DKIM-Result: none/none X-HE-Tag: 1613761146-166049 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2/19/21 1:11 AM, Michal Hocko wrote: > On Wed 17-02-21 12:41:35, Tim Chen wrote: >> Memory is accessed at a much lower frequency >> for the second cgroup. The memcg event update was not triggered for the >> second cgroup as the memcg event update didn't happened on the 1024th sample. >> The second cgroup was not placed on the soft limit tree and we didn't >> try to reclaim the excess pages. >> >> As time goes on, we saw that the first cgroup was kept close to its >> soft limit due to reclaim activities, while the second cgroup's memory >> usage slowly creep up as it keeps getting missed from the soft limit tree >> update as the update didn't fall on the modulo 1024 sample. As a result, >> the memory usage of the second cgroup keeps growing over the soft limit >> for a long time due to its relatively rare occurrence. > > Soft limit is evaluated every THRESHOLDS_EVENTS_TARGET * SOFTLIMIT_EVENTS_TARGET. > If all events correspond with a newly charged memory and the last event > was just about the soft limit boundary then we should be bound by 128k > pages (512M and much more if this were huge pages) which is a lot! > I haven't realized this was that much. Now I see the problem. This would > be a useful information for the changelog. > > Your fix is focusing on the over-the-limit boundary which will solve the > problem but wouldn't that lead to to updates happening too often in > pathological situation when a memcg would get reclaimed immediatelly? Not really immediately. The memcg that has the most soft limit excess will be chosen for page reclaim, which is the way it should be. It is less likely that a memcg that just exceeded the soft limit becomes the worst offender immediately. With the fix, we make sure that it is on the bad guys list and will not be ignored and be chosen eventually for reclaim. It will not sneakily increase its memory usage slowly. > > One way around that would be to lower the SOFTLIMIT_EVENTS_TARGET. Have > you tried that? Do we even need a separate treshold for soft limit, why > cannot we simply update the tree each MEM_CGROUP_TARGET_THRESH? > Lowering the threshold is a band aid that really doesn't fix the problem. I found that if the cgroup touches the memory infrequently enough, you could still miss the update of it. And in the mean time, you are updating things a lot more frequently with added overhead. Tim