From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6277DC77B73 for ; Thu, 27 Apr 2023 14:06:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AF7A66B0071; Thu, 27 Apr 2023 10:06:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AA7BD6B0072; Thu, 27 Apr 2023 10:06:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 947B86B0074; Thu, 27 Apr 2023 10:06:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 82C176B0071 for ; Thu, 27 Apr 2023 10:06:56 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 52A751202C0 for ; Thu, 27 Apr 2023 14:06:56 +0000 (UTC) X-FDA: 80727347232.08.4C605D6 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf20.hostedemail.com (Postfix) with ESMTP id E19781C0032 for ; Thu, 27 Apr 2023 14:06:52 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=hKsbcFXh; spf=pass (imf20.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682604413; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FSSPWlR+fmRaVisUOZECUtbamoubJx2aUQz5oMIT4bY=; b=fDUCXAVSALbd8vlV93H+dE9yAWDnncJhaFw8I5/3qaIbEIsjZ89fiWwfT1kS8k/SfNeVfl QJFxsxnxJ54I6a/hQpu+aUGEzo7eU0LRUGGjechDAje8mRtqJsvR53HQDAekvPITa6Vy39 Xov7VW9WNRkzV7jKXQ+rZ1ufv8ilLaI= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=hKsbcFXh; spf=pass (imf20.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682604413; a=rsa-sha256; cv=none; b=wIuL8OYGdaBqV26/6weyxcG/pz+x8c4oh2JrM1XwKshHwOAOpE/o9SzmN3oEut8iIwR/tp tYcC2xgajO5q9yogFzQZ13DcM5O0v0yVBkQz3I6YPkmxglDSnDyqXdi3ZZfP+TVYEAFuqK qs2qxnz18auy3bc99mXYfEhJ9a/NngM= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 61FA91FE22; Thu, 27 Apr 2023 14:06:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1682604411; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FSSPWlR+fmRaVisUOZECUtbamoubJx2aUQz5oMIT4bY=; b=hKsbcFXhcnHul4v4HNQG3MzF3hq7i1IWsZlGNpji0BcJbTTrTel8B1It9NdUZ03HwIfZuy 0abcxObKmsKVLCZiUHzjViteSgmqRQ50fjmCojQynd3A1ABQ4xVmJ+CiM4H2lEhizMKgfp IU2sZ/WkV5Q9E3dTGvzCyHzglVpEcck= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 457D6138F9; Thu, 27 Apr 2023 14:06:51 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id iBiuDnuBSmRyDgAAMHmgww (envelope-from ); Thu, 27 Apr 2023 14:06:51 +0000 Date: Thu, 27 Apr 2023 16:06:50 +0200 From: Michal Hocko To: Yosry Ahmed Cc: Johannes Weiner , Roman Gushchin , Shakeel Butt , Andrew Morton , Muchun Song , Sergey Senozhatsky , Steven Rostedt , Petr Mladek , Chris Li , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 2/2] memcg: dump memory.stat during cgroup OOM for v1 Message-ID: References: <20230426133919.1342942-1-yosryahmed@google.com> <20230426133919.1342942-3-yosryahmed@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E19781C0032 X-Stat-Signature: pugndhgyuh76j1p36wu91scuhwpsrd58 X-HE-Tag: 1682604412-7876 X-HE-Meta: U2FsdGVkX1+IoOCyODY2B7oGo8ZNiyyhkJNy/rwh+kN2vbnPZyJH0ATaS7S4JGVIC0RUKA2TRocLlj5wgF8s2rjjRB7uzuFxKYTpo4UDSLJ+OJfcAaJQAT9BrF8acFyZMUNi1InOqpSmi5spyC1G/s9qVMtp1VvpwlDkWzAB3HVbXvBnJMQ/M3Jsoz83hR6b85yDu8/3hLs10VHIeiWcvusYPC5rP0v6JceGqPV3+rrestdedLUsu4YZgmvZO1JaVkpRkp+64/M6ubtq8HZ7ppkAoXKk9WSQVJ2kNi/GPCgDvUGBQDNpEegeeUz+/iEC3wgxFe18vevXJAQ5KBhT5U8PF31kGXohhnMCOt3ApYMxgkA9dUoC5wm2KGqhpgLLpaVPmvfHoBoctCf0lPFcU0nyeb55HzXZyFzC/La14+VuykywjfJcZblnXVfF8Xg008nTg8s71j4TuMVJY6jABBEaYiRlORNlmmYLMAvuXlEkRZguYHXjOJwH79uFMx8BT2jE3dQXm54SpyGgE+uJVxle6/nz+eV6GvtiuNK9QpxLqZB4Hw2okNBZRNv9G78Mhb7r9H785v3EI7ziu2XbNerFjh8sAMJnr3rpduds9qQo5L8jbINv4RcSd5rF3uUjYAdUHPlydWVQvYihEmDtnCmkOAgI3Lip8Z6gHjQdvNU7VJinxvQ4SMfhnh8Z/hspYuuRQcHhqKctiwMaDbbZFMxM2EMf6mxDxsOVscN+2XIJbsVeX7uaZCy7F7fjzBU9DunK8oQxhTbSgf44wRc1WN003uJ62vNVyM6ddN/nvO7kbOOBaHYS7we+KCvTXfLHZQpPBuMAucsBrdnBueXftGvAIoVvkgmpXXPSZXyvdo7za4d+/gmuGi87T4/jKQaRziFO7WSDH3vhmp3bf6Vjsr2LFpKCG6UWruoC42RhBy+4PE8FByZ5T6lnixoQ+BwcCddV13mtFpsSAnHCig2 ExayEBzF xNZ5SqlkZtdzX2o17CNgU5MdPGXymJaDIWMR9VuTxwhe5rvBnaUQP5SyGmueMekHL4VtjpnDwigRBtAA5tIYXSW0C9uL8/7xFVjLy0p48HiCB867jp5NIguJ/Mef5/dW+Ej1hhhVoUTO7CF4lsi0VQ1uV7M7pBxn+ip7/cCVbS0qQuQEIe1xI+b7e64ZfWfjgrIaVAV8sF58srIxP8pAyDAWUQw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu 27-04-23 02:21:30, Yosry Ahmed wrote: > On Wed, Apr 26, 2023 at 8:27 AM Michal Hocko wrote: > > > > On Wed 26-04-23 13:39:19, Yosry Ahmed wrote: > > > Commit c8713d0b2312 ("mm: memcontrol: dump memory.stat during cgroup > > > OOM") made sure we dump all the stats in memory.stat during a cgroup > > > OOM, but it also introduced a slight behavioral change. The code used to > > > print the non-hierarchical v1 cgroup stats for the entire cgroup > > > subtree, not it only prints the v2 cgroup stats for the cgroup under > > > OOM. > > > > > > Although v2 stats are a superset of v1 stats, some of them have > > > different naming. We also lost the non-hierarchical stats for the cgroup > > > under OOM in v1. > > > > Why is that a problem worth solving? It would be also nice to add an > > example of the oom report before and after the patch. > > -- > > Michal Hocko > > SUSE Labs > > Thanks for taking a look! > > The problem is that when upgrading to a kernel that contains > c8713d0b2312 on cgroup v1, the OOM logs suddenly change. The stats > names become different, a couple of stats are gone, and the > non-hierarchical stats disappear. > > The non-hierarchical stats are important to identify if a memcg OOM'd > because of the memory consumption of its own processes or its > descendants. In the example below, I created a parent memcg "a", and a > child memcg "b". A process in "a" itself ("tail" in this case) is > hogging memory and causing an OOM, not the processes in the child "b" > (the "sleep" processes). With non-hierarchical stats, it's clear that > this is the case. Is this difference really important from the OOM POV. There is no group oom semantic in v1 and so it always boils down to a specific process that gets selected. Which memcg it is sitting in shouldn't matter all that much. Or does it really matter? > Also, it is generally nice to keep things consistent as much as > possible. The sudden change of the OOM log with the kernel upgrade is > confusing, especially that the memcg stats in the OOM logs in cgroup > v1 now look different from the stats in memory.stat. Generally speaking oom report is not carved into stone. While we shouldn't make changes just nilly willy it might change for implementation specific reasons. In this particular case I would agree that the new output is more confusing than helpful. Just look at > [ 88.339505] pgscan 0 > [ 88.339505] pgsteal 0 > [ 88.339506] pgscan_kswapd 0 > [ 88.339506] pgscan_direct 0 > [ 88.339507] pgscan_khugepaged 0 > [ 88.339507] pgsteal_kswapd 0 > [ 88.339508] pgsteal_direct 0 > [ 88.339508] pgsteal_khugepaged 0 These stats are actively misleading because it would suggest there was no memory reclaim done before oom was hit and that would imply a potentially premature OOM killer invocation (thus a bug). There are likely other stats which are not tracked in v1 yet they are reported that might add to the confusion. I believe this would be a sound justification to get back to the original reporting. -- Michal Hocko SUSE Labs