From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03266C4167B for ; Wed, 29 Nov 2023 23:12:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59BB08D0025; Wed, 29 Nov 2023 18:12:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 54BDD8D0001; Wed, 29 Nov 2023 18:12:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3EDBB8D0025; Wed, 29 Nov 2023 18:12:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2EC6D8D0001 for ; Wed, 29 Nov 2023 18:12:04 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E09311A0624 for ; Wed, 29 Nov 2023 23:12:03 +0000 (UTC) X-FDA: 81512541726.02.DE26DF4 Received: from out-171.mta1.migadu.com (out-171.mta1.migadu.com [95.215.58.171]) by imf14.hostedemail.com (Postfix) with ESMTP id CD403100019 for ; Wed, 29 Nov 2023 23:12:01 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=wcLd0b+S; spf=pass (imf14.hostedemail.com: domain of kent.overstreet@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701299522; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bz6GxSchlHkXU2aV4ba3kIBQOXF/boj29ncnlCvytHU=; b=kL+GrzbyU20tTfJF6j5tACpU50ttXBjp3c3f0pPZGh+a0qqxOupGfIUOJj6iJcqRjBjnj6 qNqvQDt1oPWOiIAlsGEmvq05TK2WTvDTMMVVz1iN/q1YPZR8Ejkw2MNgvhhNWn6NDlv/kS KfqTGUbL5RP+KVVMYkcZf8itTeHIsk0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701299522; a=rsa-sha256; cv=none; b=noPFJ0CGiiFY15dxIqHyERX4TNK/jz5kgpirI5pjzsO28VKmE0rKdkA6qq0ftxxLRyCAh3 E5UZIF1GU2EReHNHsz8HY3WTvmxY+0l5c0/rLEJD01O2tgB5veZNtd0AQ0lVA6p0TrnxjQ c/DIQzqsOdBaYxc04FC8oLJNrV1rvGE= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=wcLd0b+S; spf=pass (imf14.hostedemail.com: domain of kent.overstreet@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev Date: Wed, 29 Nov 2023 18:11:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701299520; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=bz6GxSchlHkXU2aV4ba3kIBQOXF/boj29ncnlCvytHU=; b=wcLd0b+SU3Q08M7z+7e+p3ST5xtyx3lE5TeHDRElAIXAZRCPie/oUJeFa7jSIYZP1KIFDw OEMI1w2ReC27VoEy+nvYH4raQL+GrSzwULOn1wgD//78O4uhqRkP8fE97JrBLkErScz7AF M4ouptTywkbdStIdTJ4FJRTitezve88= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Michal Hocko Cc: Roman Gushchin , Qi Zheng , Muchun Song , Linux-MM , linux-kernel@vger.kernel.org, Andrew Morton , Dave Chinner Subject: Re: [PATCH 2/7] mm: shrinker: Add a .to_text() method for shrinkers Message-ID: <20231129231147.7msiocerq7phxnyu@moria.home.lan> References: <20231122232515.177833-3-kent.overstreet@linux.dev> <20231123212411.s6r5ekvkklvhwfra@moria.home.lan> <4caadff7-1df0-45cc-9d43-e616f9e4ddb3@bytedance.com> <20231125003009.tbaxuquny43uwei3@moria.home.lan> <76A1EE85-B62C-49B3-889C-80F9A2A88040@linux.dev> <20231128035345.5c7yc7jnautjpfoc@moria.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Stat-Signature: jdzxy73znomrop6g8mtofuymo86huz3h X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: CD403100019 X-Rspam-User: X-HE-Tag: 1701299521-871621 X-HE-Meta: U2FsdGVkX1+bWNIFB6CS/tItbq6w407s81S1qZSXZv+lHHTUG4zdba7DTQSLozxt+oRDjj7kNZWJCjCG7xuUF2bgl6ePZUVlVBcVY6vEHLqP0nKBb4DPm5q42Jx4G2nuoO811H39RuUlmiuDc5FpiHdFifdLM/yJTF3/5mUuSZCg6SgT0N/7d+rtvW6Smb/rLgzxPJRguEGwknMRwdWt0PSB/Hwa1cK4xM/XKiozHtjxXznPwyLbQRlkttkfcLR3lWNmvRMDD1zUHuebTxVRsC4FUF1UvslZXeelt1xQBXKQos9gkHP/t3rE22S/FWJ+uRfycr7r0rtKW3xv/ABSVRW3bFImQT7JB3kL19HnuWejJ+8IHtuldFHTVXm+RwGlgpYqdVaiEsBEOzM5CDhW5U6IVEzRee/onOxzzR2pCvaazpbtApAP0/0gCczf4VAfKuLzyvBweBzgZ/XVSHJsItLNNzmZrVldGLcSsll081K7TfDQXZJhNB4+r/wa6iA0Dv5zMLx8qvfWeNTVf6CNWHiQ9W7+pSyWdBFiHKCQ9Ls/2sTuP4b+NA4/lXHVZ6jXPXEuMem7E0sxskllapBSGedDNqXx9tmel84TO8pcb/TYXKXKCwMgdGGCJvLq+QCPvi9Ofxk0sH/59VTe0GZHFWC7PggwR1yu8/VHRufs1EscekSNDq2VBHeM+wUBmgHPe1RSVA7ILgDUB7eUmvyJz3btXNvMHxeuKKPnloAHulhQylOXEDbWIS02Ep/7KDLp9fBc6hXOj/ZQpM4V49tI9M8/CufK/JdBLXKW8hZinQC4x95CoJUQVCa2vaXW1/n4je7MZ2Ocymu68MiOJdjcfEH4Zvh9fkMLKiwGth99jeLzMVEVuX/GuZ7xb0uyuwFpJ7pF6suYMOrd9L1uGqflNkUjM4xWKLZdcvXZtKGJn23tOLtj5TwDEyVPVW1UC3lHh4rVLyp0JYVtBdFERe0 AWChkpXZ AcQNvSmF4xHMiazYqdCpcvwD2K5qhpM1EsJr9Ff3/LjSLac0LZTTvAIRQIzqmIvyNmeERYJTKoRZZVK/wMpvYAJ8P8Iwhq+rntdRs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Nov 29, 2023 at 10:14:54AM +0100, Michal Hocko wrote: > On Tue 28-11-23 16:34:35, Roman Gushchin wrote: > > On Tue, Nov 28, 2023 at 02:23:36PM +0800, Qi Zheng wrote: > [...] > > > Now I think adding this method might not be a good idea. If we allow > > > shrinkers to report thier own private information, OOM logs may become > > > cluttered. Most people only care about some general information when > > > troubleshooting OOM problem, but not the private information of a > > > shrinker. > > > > I agree with that. > > > > It seems that the feature is mostly useful for kernel developers and it's easily > > achievable by attaching a bpf program to the oom handler. If it requires a bit > > of work on the bpf side, we can do that instead, but probably not. And this > > solution can potentially provide way more information in a more flexible way. > > > > So I'm not convinced it's a good idea to make the generic oom handling code > > more complicated and fragile for everybody, as well as making oom reports differ > > more between kernel versions and configurations. > > Completely agreed! From my many years of experience of oom reports > analysing from production systems I would conclude the following categories > - clear runaways (and/or memory leaks) > - userspace consumers - either shmem or anonymous memory > predominantly consumes the memory, swap is either depleted > or not configured. > OOM report is usually useful to pinpoint those as we > have required counters available > - kernel memory consumers - if we are lucky they are > using slab allocator and unreclaimable slab is a huge > part of the memory consumption. If this is a page > allocator user the oom repport only helps to deduce > the fact by looking at how much user + slab + page > table etc. form. But identifying the root cause is > close to impossible without something like page_owner > or a crash dump. > - misbehaving memory reclaim > - minority of issues and the oom report is usually > insufficient to drill down to the root cause. If the > problem is reproducible then collecting vmstat data > can give a much better clue. > - high number of slab reclaimable objects or free swap > are good indicators. Shrinkers data could be > potentially helpful in the slab case but I really have > hard time to remember any such situation. > On non-production systems the situation is quite different. I can see > how it could be very beneficial to add a very specific debugging data > for subsystem/shrinker which is developed and could cause the OOM. For > that purpose the proposed scheme is rather inflexible AFAICS. Considering that you're an MM guy, and that shrinkers are pretty much universally used by _filesystem_ people - I'm not sure your experience is the most relevant here? The general attitude I've been seeing in this thread has been one of dismissiveness towards filesystem people. Roman too; back when he was working on his shrinker debug feature I reached out to him, explained that I was working on my own, and asked about collaborating - got crickets in response... Hmm.. Besides that, I haven't seen anything what-so-ever out of you guys to make our lives easier, regarding OOM debugging, nor do you guys even seem interested in the needs and perspectives of the filesytem people. Roman, your feature didn't help one bit for OOM debuging - didn't even come with documentation or hints as to what it's for. BPF? Please. Anyways. Regarding log spam: that's something this patchset already starts to address. I don't think we needed to be dumping every single slab in the system, for ~2 pages worth of logs; hence this patchset changes that to just print the top 10. The same approach is taken with shrinkers: more targeted, less spammy output. So now that that concern has been addressed, perhaps some actual meat: For one, the patchset adds tracking for when a shrinker was last asked to free something, vs. when it was actually freed. So right there, we can finally see at a glance when a shrinker has gotten stuck and which one. Next up, why has a shrinker gotten stuck? That's why the .to_text() callback is needed: _shrinkers have internal state_, and the reasons objects may not be getting freed are specific to a given shrinker implementation. In bcachefs we added counters for each individual reason an object may be skipped by the shrinker (io in flight? that debugged a runaway prefetch issue. too many dirty? that points to journal reclaim). I'm working on a .to_text() function for the struct super_block shrinker, will post that one soon...