From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 706CFC4167B for ; Fri, 1 Dec 2023 20:01:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 08A0C6B03AA; Fri, 1 Dec 2023 15:01:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 014576B03DC; Fri, 1 Dec 2023 15:01:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF7476B03E1; Fri, 1 Dec 2023 15:01:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CA2C26B03AA for ; Fri, 1 Dec 2023 15:01:43 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9D843A06A4 for ; Fri, 1 Dec 2023 20:01:43 +0000 (UTC) X-FDA: 81519319686.11.6036E4A Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) by imf24.hostedemail.com (Postfix) with ESMTP id 53B42180030 for ; Fri, 1 Dec 2023 20:01:40 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=imM9s+0U; spf=pass (imf24.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701460901; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L4EsUjCpDqR5ukRbUrdGLVdW+brLvnqNkoJINxDTUj4=; b=P30XT0T1LxEcZWBMmbyEcWLL//V95AXVvrefO4LhRtU4cebj/w1aSs1IviaC4SHPuPGLA7 5wVGVpNHLq6wbFVBZlaMqGXcyHpB39gE+/xlkVUaKqKSbOOeID+CRyCtXNbeYPWNV5yZif zXUKLly9Mun6YZdnN9HTUlt+x91kF6M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701460901; a=rsa-sha256; cv=none; b=OGG3MZjgZPhNODgv55R77UIwPKTDl8PlBdAVpDRQ8ifDV/XQQGKPk/NP+wctgmDQ/mtzv/ nRJlr2I3NvCx8JGwinOSm3CT8TNGICfPLFMKavSWMGVukoD0fKp600wZ1fTWuSzJX/BeRQ lIa6tE7TS7yOJxTHOhKMzatcu+g7py4= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=imM9s+0U; spf=pass (imf24.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.170 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev Date: Fri, 1 Dec 2023 12:01:33 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701460898; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=L4EsUjCpDqR5ukRbUrdGLVdW+brLvnqNkoJINxDTUj4=; b=imM9s+0UUfzimqJ8u6k96q03kHMIRBQOgk5f45b7nxNLxLWwGy/XhsfjLsd4SIikXhSdPg Mx7ovIdRCZz9QgY6LMils28s6R6uBRdlmfUCTISH4vw315Z0la9owPgNYDmBqfpGn9GpKc 5KZwnG40OoqDZPcOv/k5NWqbO9oSEOA= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Roman Gushchin To: Dave Chinner Cc: Kent Overstreet , Qi Zheng , Michal Hocko , Muchun Song , Linux-MM , linux-kernel@vger.kernel.org, Andrew Morton Subject: Re: [PATCH 2/7] mm: shrinker: Add a .to_text() method for shrinkers Message-ID: References: <76A1EE85-B62C-49B3-889C-80F9A2A88040@linux.dev> <20231128035345.5c7yc7jnautjpfoc@moria.home.lan> <20231129231147.7msiocerq7phxnyu@moria.home.lan> <04f63966-af72-43ef-a65c-ff927064a3e4@bytedance.com> <20231130032149.ynap4ai47dj62fy3@moria.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 53B42180030 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: f568nzucyikbat8qrzdi8787oh5oi9xc X-HE-Tag: 1701460900-21274 X-HE-Meta: U2FsdGVkX18KIHJ8KxcUOafAjji1cAAcRwr4kORtUGPn7lUq0+oayvIoXE8ldRXzEheQwCKHDKC3UN9zyNf9Ow9zW1GNFLa5Tbhe+VPK/jTdBMu00NsO7UkQODfRvjoyprTraV6rRBmpr2FEERKzdwghC16e6RdD+xf72pT0Xdhh/U0+m44Q5bSpi6IWoEWKPiQOezZXIzlcak/hcAZbFlJk5e0uMqOLaDRWe3CpPuq9WUhAIrRQ5maBpv+4z+jG5BnOAOCoiwwqm59O991eaHaZETq6X3Pu9lVpCSqkmxZ1F9n+OimxkKPHR/I4hj7hLCJUIlS7qae6CKOyxssx+W+ea4SFTSXtUBIYikAeom0ojrHLggXlLRs/VGKOY+EUdLKlX2r1kP30yHh3I2RLQJ97SqUZuhi4wjIvQUxEZQh/FsYaKWi2cxSR+OfpY6rQBboumWGbWX+kLhV90uH4cah3OVtTEDCCLL00abDpbZN9m2Qn31uuitE/fsGK9PqhzoH57/76kYM1FWtwnZlAIpiesDTBomdwVMihcqCJ9iXTf7MJlwkLgSe6zQi+yV4s4LCYmZKvlWvRjicz4L1dLIKTTsxGNT6XyyALVw4fBMB/VMvCGuy1gsBg49hSTECeHRlevtZa6sj84OH3rZ6y0afEN4CFqoa8hA+F2uQW0K+/E2ndgDpTG4zrNbkwje4lWXn/z2RcC9BMTqASEFKdCvm/Rwwc5hLKA7Uq39DhWrg4LI9tTP1eceKpmwF4lJxeayc83+XMg+ekBpw7YswHIkrPBMXc8bGB7ySQvSBWZtzZH/n1BCkYd4mXbuXO0dRE6V8QD3reLNDMDbouwccN9/3BzI450099PQjJospbKD/UICNmXTRys4vEN2HfWFqZo4E8Nv8ZRscIRatqVGGCFimv4JuhNh1CD0UqWcwgMY8fXtevT9KUPYqg8VU56UqdPZhLAQrm78pzQqGRGsZ NSH4CWnX F6jxIVsFZnNv0e7YoiQj/+YxlVCTTBGrQ/I/y3mBVAVn6KktQrvwUwaz/KMLOb+M44nMwvs4u0GL/CZ/0wgFqw1k1OTCVSA1Z4UjZbV35OlARbaIesPdw9K+Dp/WA0O4flwUwMg6CEW7h0e/Pt31qqIcrs7y002R2w8121EImt3uWNFP+iwazhvNgTw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Dec 01, 2023 at 12:18:44PM +1100, Dave Chinner wrote: > On Thu, Nov 30, 2023 at 11:01:23AM -0800, Roman Gushchin wrote: > > On Wed, Nov 29, 2023 at 10:21:49PM -0500, Kent Overstreet wrote: > > > On Thu, Nov 30, 2023 at 11:09:42AM +0800, Qi Zheng wrote: > > > > For non-bcachefs developers, who knows what those statistics mean? > > > Ok, a simple question then: > > why can't you dump /proc/slabinfo after the OOM? > > Taken to it's logical conclusion, we arrive at: > > OOM-kill doesn't need to output anything at all except for > what it killed because we can dump > /proc/{mem,zone,vmalloc,buddy,slab}info after the OOM.... > > As it is, even asking such a question shows that you haven't looked > at the OOM kill output for a long time - it already reports the slab > cache usage information for caches that are reclaimable. > > That is, if too much accounted slab cache based memory consumption > is detected at OOM-kill, it will calldump_unreclaimable_slab() to > dump all the SLAB_RECLAIM_ACCOUNT caches (i.e. those with shrinkers) > to the console as part of the OOM-kill output. You are right, I missed that, partially because most of OOM's I had to deal with recently were memcg OOM's. This changes my perspective at Kent's patches, if we dump this information already, it might be not a bad idea to do it nicer. So I take my words back here. > > The problem Kent is trying to address is that this output *isn't > sufficient to debug shrinker based memory reclaim issues*. It hasn't > been for a long time, and so we've all got our own special debug > patches and methods for checking that shrinkers are doing what they > are supposed to. Kent is trying to formalise one of the more useful > general methods for exposing that internal information when OOM > occurs... > > Indeed, I can think of several uses for a shrinker->to_text() output > that we simply cannot do right now. > > Any shrinker that does garbage collection on something that is not a > pure slab cache (e.g. xfs buffer cache, xfs inode gc subsystem, > graphics memory allocators, binder, etc) has no visibility of the > actuall memory being used by the subsystem in the OOM-kill output. > This information isn't in /proc/slabinfo, it's not accounted by a > SLAB_RECLAIM_ACCOUNT cache, and it's not accounted by anything in > the core mm statistics. > > e.g. How does anyone other than a XFS expert know that the 500k of > active xfs_buf handles in the slab cache actually pins 15GB of > cached metadata allocated directly from the page allocator, not just > the 150MB of slab cache the handles take up? > > Another example is that an inode can pin lots of heap memory (e.g. > for in-memory extent lists) and that may not be freeable until the > inode is reclaimed. So while the slab cache might not be excesively > large, we might have an a million inodes with a billion cumulative > extents cached in memory and it is the heap memory consumed by the > cached extents that is consuming the 30GB of "missing" kernel memory > that is causing OOM-kills to occur. > > How is a user or developer supposed to know when one of these > situations has occurred given the current lack of memory usage > introspection into subsystems? What would be the proper solution to this problem from your point of view? What functionality/API mm can provide to make the life of fs developers better here? > > These are the sorts of situations that shrinker->to_text() would > allow us to enumerate when it is necessary (i.e. at OOM-kill). At > any other time, it just doesn't matter, but when we're at OOM having > a mechanism to report somewhat accurate subsystem memory consumption > would be very useful indeed. > > > Unlike anon memory, slab memory (fs caches in particular) should not be heavily > > affected by killing some userspace task. > > Whether tasks get killed or not is completely irrelevant. The issue > is that not all memory that is reclaimed by shrinkers is either pure > slab cache memory or directly accounted as reclaimable to the mm > subsystem.... My problem with the current OOM reporting infrastructure (and it's a bit an offtopic here) - it's good for manually looking into these reports, but not particularly great for automatic collection and analysis at scale. So this is where I was coming from. Thanks!