From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A04BC432C0 for ; Wed, 20 Nov 2019 11:40:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1254822419 for ; Wed, 20 Nov 2019 11:40:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1254822419 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 667736B026B; Wed, 20 Nov 2019 06:40:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 617446B026C; Wed, 20 Nov 2019 06:40:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52D716B026D; Wed, 20 Nov 2019 06:40:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id 3C1576B026B for ; Wed, 20 Nov 2019 06:40:47 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id C90AF181AEF1A for ; Wed, 20 Nov 2019 11:40:46 +0000 (UTC) X-FDA: 76176463692.01.knot32_6bda30e80162 X-HE-Tag: knot32_6bda30e80162 X-Filterd-Recvd-Size: 6914 Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Wed, 20 Nov 2019 11:40:46 +0000 (UTC) Received: by mail-wr1-f65.google.com with SMTP id t1so27741949wrv.4 for ; Wed, 20 Nov 2019 03:40:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=bUzs01x7ltpYKKYcHWi+QFWBDEgdcUpOqE20nPEwSYo=; b=SapDwCTmppRV9xFxJ/GJ4ouVN2B6fO/Usg0gXTgFGqVSOdSDBbaBcAMaTvnw4Bgv1L OKgSevzvLSVsVv1BlBeJgQtkajlNFKygE3ZUwREUg3F6bng+5We2fDUDrNiRRv93ZT1N Qwuv3b9nn+uMn6NgWw6b4C9JbDVURgrQ6qOsQDBYWP5VyYNKVuH8XIM+CxkhK02f4gnm Lac2XxaQLEs9EclbDJ30RbrfprxrtXHq8Ioqoh8R79EjFKbwjZTDPcu+bZ6uL+YpsR8e /DoCnIpjHZRZIXlih3rc73OBtjx2BsYl8L/XTSUJzMk4RkJZGeYGZl9nsHlPwvXP4tui A/LA== X-Gm-Message-State: APjAAAXBmgyEjmDkCFi4ssyX9KpemwkTeb4FfwRJULymhNoZ7zOAJ5bs VUn0KVxnlwP3WWGxkyaoNkA= X-Google-Smtp-Source: APXvYqyxW+yqbix570d0xv1yywya6zcSss0c03RlZsXw8WfUe0YfkIAmEk8HSkg2jpw2j2lYjshMBA== X-Received: by 2002:adf:a551:: with SMTP id j17mr1991567wrb.18.1574250045044; Wed, 20 Nov 2019 03:40:45 -0800 (PST) Received: from localhost (prg-ext-pat.suse.com. [213.151.95.130]) by smtp.gmail.com with ESMTPSA id b3sm6081583wmj.44.2019.11.20.03.40.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Nov 2019 03:40:44 -0800 (PST) Date: Wed, 20 Nov 2019 12:40:43 +0100 From: Michal Hocko To: Yafang Shao Cc: Johannes Weiner , Vladimir Davydov , Andrew Morton , Linux MM Subject: Re: [PATCH] mm, memcg: show memcg min setting in oom messages Message-ID: <20191120114043.GH23213@dhcp22.suse.cz> References: <1574239985-1916-1-git-send-email-laoar.shao@gmail.com> <20191120102157.GF23213@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 20-11-19 18:53:44, Yafang Shao wrote: > On Wed, Nov 20, 2019 at 6:22 PM Michal Hocko wrote: > > > > On Wed 20-11-19 03:53:05, Yafang Shao wrote: > > > A task running in a memcg may OOM because of the memory.min settings of his > > > slibing and parent. If this happens, the current oom messages can't show > > > why file page cache can't be reclaimed. > > > > min limit is not the only way to protect memory from being reclaim. The > > memory might be pinned or unreclaimable for other reasons (e.g. swap > > quota exceeded for memcg). > > Both swap or unreclaimabed (unevicteable) is printed in OOM messages. Not really. Consider a memcg which has reached it's swap limit. The anonymous memory is not really reclaimable even when there is a lot of swap space available. > If something else can prevent the file cache being reclaimed, we'd > better show them as well. How are you going to do that? How do you track pins on pages? > > Besides that, there is the very same problem > > with the global OOM killer, right? And I do not expect we want to print > > all memcgs in the system (this might be hundreds). > > > > I forgot the global oom... > > Why not just print the memcgs which are under memory.min protection or > something like a total number of min protected memory ? Yes, this would likely help. But the main question really reamains, is this really worth it? > > > So it is better to show the memcg > > > min settings. > > > Let's take an example. > > > bar bar/memory.max = 1200M memory.min=800M > > > / \ > > > barA barB barA/memory.min = 800M memory.current=1G (file page cache) > > > barB/memory.min = 0 (process in this memcg is allocating page) > > > > > > The process will do memcg reclaim if the bar/memory.max is reached. Once > > > the barA/memory.min is reached it will stop reclaiming file page caches in > > > barA, and if there is no reclaimable pages in bar and bar/barB it will > > > enter memcg OOM then. > > > After this pacch, bellow messages will be show then (only includeing the > > > relevant messages here). The lines begin with '#' are newly added info (the > > > '#' symbol is not in the original messages). > > > memory: usage 1228800kB, limit 1228800kB, failcnt 18337 > > > ... > > > # Memory cgroup min setting: > > > # /bar: min 819200KB emin 0KB > > > # /bar/barA: min 819200KB emin 819200KB > > > # /bar/barB: min 0KB emin 0KB > > > ... > > > Memory cgroup stats for /bar: > > > anon 418328576 > > > file 835756032 > > > ... > > > unevictable 0 > > > ... > > > oom-kill:constraint=CONSTRAINT_MEMCG..oom_memcg=/bar,task_memcg=/bar/barB > > > > > > With the new added information, we can find the memory.min in bar/barA is > > > reached and the processes in bar/barB can't reclaim file page cache from > > > bar/barA any more. While without this new added information we don't know > > > why the file page cache in bar can't be reclaimed. > > > > Well, I am not sure this is really usefull enough TBH. It doesn't give > > you the whole picture and it potentially generates a lot of output in > > the oom report. FYI we used to have a more precise break down of > > counters in memcg hierarchy, see 58cf188ed649 ("memcg, oom: provide more > > precise dump info while memcg oom happening") which later got rewritten > > by c8713d0b2312 ("mm: memcontrol: dump memory.stat during cgroup OOM") > > > > At least we'd better print a total protected memory in the oom messages. > > > Could you be more specific why do you really need this piece of > > information? > > I have said in the commit log, that we don't know why the file cache > can't be reclaimed (when evictable is 0 and dirty is 0 as well.) And the counter argument is that this will not help you there much in many large and much more common cases. I argue, and I might be wrong here so feel free to correct me, that the reclaim protection guarantee (min) is something to be under admins control. It shouldn't really happen nilly-willy because it has really large consequences, the OOM including. So if there is a suspicious amount of memory that could be reclaimed normally then the reclaim protection is really the first suspect to go after. -- Michal Hocko SUSE Labs