From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5E37C433F5 for ; Mon, 31 Jan 2022 18:25:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F97C6B00DF; Mon, 31 Jan 2022 13:25:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A83B6B00E0; Mon, 31 Jan 2022 13:25:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1703C6B00E1; Mon, 31 Jan 2022 13:25:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0032.hostedemail.com [216.40.44.32]) by kanga.kvack.org (Postfix) with ESMTP id 060E06B00DF for ; Mon, 31 Jan 2022 13:25:28 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id C3C6C8E2D8 for ; Mon, 31 Jan 2022 18:25:27 +0000 (UTC) X-FDA: 79091409894.22.BC56546 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf15.hostedemail.com (Postfix) with ESMTP id 3C866A0002 for ; Mon, 31 Jan 2022 18:25:27 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id C0DA41F380; Mon, 31 Jan 2022 18:25:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1643653525; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=FTFA2b8ZYqmua0W1dg2oBKqfoda+M3kBiT0mJekqZsw=; b=XjVzr1CnsL27EfZQj2F2hpCP5vbc4QBXNF1eoFWrbyL2VFCRPnJydl3ZVbHSVp5xUVA7lF QwQr3bMoRIBAy7byOQcMSGSh7g7wv07HDD4L3O9V/0Obawh8FZcQ+JUekpCtFpTdW3gypp sBg/1U1rWLah6wt4shoX2WxFPps9PYo= Received: from suse.cz (unknown [10.100.201.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 4497AA3B8E; Mon, 31 Jan 2022 18:25:25 +0000 (UTC) Date: Mon, 31 Jan 2022 19:25:22 +0100 From: Michal Hocko To: Roman Gushchin Cc: Johannes Weiner , Waiman Long , Vladimir Davydov , Andrew Morton , Petr Mladek , Steven Rostedt , Sergey Senozhatsky , Andy Shevchenko , Rasmus Villemoes , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Ira Weiny , Rafael Aquini Subject: Re: [PATCH v2 3/3] mm/page_owner: Dump memcg information Message-ID: References: <20220129205315.478628-1-longman@redhat.com> <20220129205315.478628-4-longman@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3C866A0002 X-Stat-Signature: joe9hdgjznbmbjz3mr3cxesbgjif9ex3 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=XjVzr1Cn; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf15.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com X-Rspam-User: nil X-HE-Tag: 1643653527-699097 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon 31-01-22 10:15:45, Roman Gushchin wrote: > On Mon, Jan 31, 2022 at 11:53:19AM -0500, Johannes Weiner wrote: > > On Mon, Jan 31, 2022 at 10:38:51AM +0100, Michal Hocko wrote: > > > On Sat 29-01-22 15:53:15, Waiman Long wrote: > > > > It was found that a number of offlined memcgs were not freed because > > > > they were pinned by some charged pages that were present. Even "echo > > > > 1 > /proc/sys/vm/drop_caches" wasn't able to free those pages. These > > > > offlined but not freed memcgs tend to increase in number over time with > > > > the side effect that percpu memory consumption as shown in /proc/meminfo > > > > also increases over time. > > > > > > > > In order to find out more information about those pages that pin > > > > offlined memcgs, the page_owner feature is extended to dump memory > > > > cgroup information especially whether the cgroup is offlined or not. > > > > > > It is not really clear to me how this is supposed to be used. Are you > > > really dumping all the pages in the system to find out offline memcgs? > > > That looks rather clumsy to me. I am not against adding memcg > > > information to the page owner output. That can be useful in other > > > contexts. > > > > We've sometimes done exactly that in production, but with drgn > > scripts. It's not very common, so it doesn't need to be very efficient > > either. Typically, we'd encounter a host with an unusual number of > > dying cgroups, ssh in and poke around with drgn to figure out what > > kind of objects are still pinning the cgroups in question. > > > > This patch would make that process a little easier, I suppose. > > Right. Over last few years I've spent enormous amount of time digging into > various aspects of this problem and in my experience the combination of drgn > for the inspection of the current state and bpf for following various decisions > on the reclaim path was the most useful combination. > > I really appreciate an effort to put useful tools to track memcg references > into the kernel tree, however the page_owner infra has a limited usefulness > as it has to be enabled on the boot. But because it doesn't add any overhead, > I also don't think there any reasons to not add it. Would it be feasible to add a debugfs interface to displa dead memcg information? -- Michal Hocko SUSE Labs