From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD603C5519F for ; Wed, 25 Nov 2020 13:37:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 06ADE206E5 for ; Wed, 25 Nov 2020 13:37:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=suse.com header.i=@suse.com header.b="KRSsdr+N" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 06ADE206E5 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=suse.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8BBFE6B0074; Wed, 25 Nov 2020 08:37:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 733D96B007E; Wed, 25 Nov 2020 08:37:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E66E6B0075; Wed, 25 Nov 2020 08:37:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0054.hostedemail.com [216.40.44.54]) by kanga.kvack.org (Postfix) with ESMTP id 293856B0073 for ; Wed, 25 Nov 2020 08:37:43 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id E6726180AD811 for ; Wed, 25 Nov 2020 13:37:42 +0000 (UTC) X-FDA: 77523043164.05.horn26_44167dd27377 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id C8EE91803DCE9 for ; Wed, 25 Nov 2020 13:37:42 +0000 (UTC) X-HE-Tag: horn26_44167dd27377 X-Filterd-Recvd-Size: 3163 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Wed, 25 Nov 2020 13:37:42 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1606311461; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1g24Rgp6aqa6Ka3pxmGa5G0dKc6hKcPeknu8BPGfcI4=; b=KRSsdr+NVduHNlWpHMcDky+6b+1SginL8vAJp2N/Syqhw+rPnjhccnUhw2OXVNSc7mt9nj GrNUI1h/tfiB0lpQjhRgGBhZZbEQjaAfXwHG7o+Oc9Bkb48GbOmjjD9qK6ix095ZgeYA9F IvWKwuwQHAcFIfsj47r3r1NCkeJOghQ= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 47CF4AC23; Wed, 25 Nov 2020 13:37:41 +0000 (UTC) Date: Wed, 25 Nov 2020 14:37:40 +0100 From: Michal Hocko To: Bruno =?iso-8859-1?Q?Pr=E9mont?= Cc: Yafang Shao , Chris Down , Johannes Weiner , cgroups@vger.kernel.org, linux-mm@kvack.org, Vladimir Davydov Subject: Re: Regression from 5.7.17 to 5.9.9 with memory.low cgroup constraints Message-ID: <20201125133740.GE31550@dhcp22.suse.cz> References: <20201125123956.61d9e16a@hemera> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <20201125123956.61d9e16a@hemera> Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, thanks for the detailed report. On Wed 25-11-20 12:39:56, Bruno Pr=E9mont wrote: [...] > Did memory.low meaning change between 5.7 and 5.9? The latest semantic change in the low limit protection semantic was introduced in 5.7 (recursive protection) but it requires an explicit enablinig. > From behavior it > feels as if inodes are not accounted to cgroup at all and kernel pushes > cgroups down to their memory.low by killing file cache if there is not > enough free memory to hold all promises (and not only when a cgroup > tries to use up to its promised amount of memory). Your counters indeed show that the low protection has been breached, most likely because the reclaim couldn't make any progress. Considering that this is the case for all/most of your cgroups it suggests that the memory pressure was global rather than limit imposed. In fact even top level cgroups got reclaimed below the low limit. This suggests that this is not likely to be memcg specific. It is more likely that this is a general memory reclaim regression for your workload. There were larger changes in that area. Be it lru balancing based on cost model by Johannes or working set tracking for anonymous pages by Joonsoo. Maybe even more. Both of them can influence page cache reclaim but you are suggesting that slab accounted memory is not reclaimed properly. I am not sure sure there were considerable changes there. Would it be possible to collect /prov/vmstat as well? --=20 Michal Hocko SUSE Labs