From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4A9FC46CA3 for ; Thu, 30 Nov 2023 08:14:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2EFFB6B0448; Thu, 30 Nov 2023 03:14:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 29F376B0449; Thu, 30 Nov 2023 03:14:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1675A6B044A; Thu, 30 Nov 2023 03:14:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 079BF6B0448 for ; Thu, 30 Nov 2023 03:14:45 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CE3E440123 for ; Thu, 30 Nov 2023 08:14:44 +0000 (UTC) X-FDA: 81513909288.10.5392A51 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf28.hostedemail.com (Postfix) with ESMTP id 9E5BBC0010 for ; Thu, 30 Nov 2023 08:14:42 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf28.hostedemail.com: domain of mhocko@suse.com designates 195.135.223.130 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701332083; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tLcNagkjKoWDAXT9DfIt7WY731/gtynNv+fjSnpK9l8=; b=DVK2ZGiT6EfzR2nQ2M+jwdHlg5RjSVJYMB5lIqcyFsRZPXJqwhO+bLAC46Ct2YuCO5tP5S mbDwAMvmegzo7DiNoLp/lz5RH/Jxqy6TZt4ciySq23Zrc9jCrxcaJmIq/S6CO8muLjzEeg ztukiOsdsMU85wRXC0TKI/EAdflTo08= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=suse.com; spf=pass (imf28.hostedemail.com: domain of mhocko@suse.com designates 195.135.223.130 as permitted sender) smtp.mailfrom=mhocko@suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701332083; a=rsa-sha256; cv=none; b=aLBadawtn9TIHGZDZmrY5qrV/KYtI/GV5oTQLS+ZX3NNWJYtmpgKtJqnLqibwdNoMUzFbb vlboiK06eg8A3r56TS+d9VdIlIDGhJ6Evoz1KJ5T365gt3n8/DZOP8BRJt1b0EszPa5FC1 O9+NuipfUMEHqqnEMo3g7WBC13zBp2s= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B8D2F21AA3; Thu, 30 Nov 2023 08:14:40 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 9A8E21342E; Thu, 30 Nov 2023 08:14:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id yeW3InBEaGUZaQAAD6G6ig (envelope-from ); Thu, 30 Nov 2023 08:14:40 +0000 Date: Thu, 30 Nov 2023 09:14:35 +0100 From: Michal Hocko To: Kent Overstreet Cc: Roman Gushchin , Qi Zheng , Muchun Song , Linux-MM , linux-kernel@vger.kernel.org, Andrew Morton , Dave Chinner Subject: Re: [PATCH 2/7] mm: shrinker: Add a .to_text() method for shrinkers Message-ID: References: <20231123212411.s6r5ekvkklvhwfra@moria.home.lan> <4caadff7-1df0-45cc-9d43-e616f9e4ddb3@bytedance.com> <20231125003009.tbaxuquny43uwei3@moria.home.lan> <76A1EE85-B62C-49B3-889C-80F9A2A88040@linux.dev> <20231128035345.5c7yc7jnautjpfoc@moria.home.lan> <20231129231147.7msiocerq7phxnyu@moria.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231129231147.7msiocerq7phxnyu@moria.home.lan> X-Rspamd-Queue-Id: 9E5BBC0010 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: h34faseddz3taw8smzog4jcyufrszzr1 X-HE-Tag: 1701332082-395583 X-HE-Meta: U2FsdGVkX19KLSnKtFheEwITdLOaLOzV26gfRI2Y86LaQ52gwyrNBqUJIC+wPKY5MM1QCgPEvWX3TRUx1SWvfrMtvUIN/ffkf8WB8GUyhlTKmhvpIr70Hgb2hbXO9CW7VfDy0aeGOqIyHpxoFkEB8hrQLc6iXxdNqZMquXuTkx97o8qBT83N8UwFAkwRtnS1E1F9IguTPxoOD79H+lG932NisqvyR4NX/MzFUUIzWKAGTK+OqLUe5Vh/PXUsvJbq2R29CHUliBc5vBLRd6ygQ3NhW2GIIgikwu4m4aLO+O/1ho8gIubK3ffN3w2moWFJb4ah+xCTSpUwMRNzKETfM4PIs2gcNe6vxYRm1JxZ0sKuc+IFiBJt1twb3ASSWYoKvHo6V2KrhNhnTc4W+abRtML5mzvcYgEZwDa3stUzQi3yNhf85OM6XbX2vEHeERDSzLKBDdXZEhG3kSDUa6xABjMyiOWPmD6OPMKIDBidnUoPZEr9vnfYyDSABhauQ8I56ezDlHyEHgrAgzRmbNloU3ErzK5gbwHFKTwEk3ANhvonct7EhklGi5SNZYQlECnwfHuz5JHX4gidCP7wy61X2XZq6BMHc434nWlllyweLFkdhk2fMCHaNWGYGI0t6O4j8PCSNWlcqhMuajkm55fPswm3YufrcliHhIzTa3drEsH0b/QfknuRaCB6vn5bsvhyEy9mL7tTjh2F9lvY3juRqVxwvatlpU28bqfNvH6Wi2dhn+wdT/URKQSxSlsTfnWCKKeUByyosiiHjKWNhtoaFrHH7xePOz0/AFEAIr9qDWRLHOcJZGBWPcAPg8g0gy17+3eoh7aigjT8CXSFXOAv3h+scvT7klzV3ipVgkw4OP7VMWReKCO+m5YTHsP2J8zeKSM6ag8CLf5Qkzmpdzi5BxbwmWYoSoY0ADp9yZg258oT2IjWeUucmsRGMNT0pPIYv2XMjaDjVJYfXOIoMoj GCiXIGEm qe8ylnGfix+MB7bfAorXJicD6ZX1ypYHboSwhrwIVhpWdj2rs1AoyVImF+MU6sL1uUYUJNZZStFxjHan5UwCcicEK6Y0lR4cwzHnaB61hUd/k7ctSka5zqEOBLa8zp7yIutwfuSAYNuIL4FEkcUVoyeml0213farr6DyDYCE8yRGiUQxAP/hcuwt+ot1DFzpdt7iE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed 29-11-23 18:11:47, Kent Overstreet wrote: > On Wed, Nov 29, 2023 at 10:14:54AM +0100, Michal Hocko wrote: > > On Tue 28-11-23 16:34:35, Roman Gushchin wrote: > > > On Tue, Nov 28, 2023 at 02:23:36PM +0800, Qi Zheng wrote: > > [...] > > > > Now I think adding this method might not be a good idea. If we allow > > > > shrinkers to report thier own private information, OOM logs may become > > > > cluttered. Most people only care about some general information when > > > > troubleshooting OOM problem, but not the private information of a > > > > shrinker. > > > > > > I agree with that. > > > > > > It seems that the feature is mostly useful for kernel developers and it's easily > > > achievable by attaching a bpf program to the oom handler. If it requires a bit > > > of work on the bpf side, we can do that instead, but probably not. And this > > > solution can potentially provide way more information in a more flexible way. > > > > > > So I'm not convinced it's a good idea to make the generic oom handling code > > > more complicated and fragile for everybody, as well as making oom reports differ > > > more between kernel versions and configurations. > > > > Completely agreed! From my many years of experience of oom reports > > analysing from production systems I would conclude the following categories > > - clear runaways (and/or memory leaks) > > - userspace consumers - either shmem or anonymous memory > > predominantly consumes the memory, swap is either depleted > > or not configured. > > OOM report is usually useful to pinpoint those as we > > have required counters available > > - kernel memory consumers - if we are lucky they are > > using slab allocator and unreclaimable slab is a huge > > part of the memory consumption. If this is a page > > allocator user the oom repport only helps to deduce > > the fact by looking at how much user + slab + page > > table etc. form. But identifying the root cause is > > close to impossible without something like page_owner > > or a crash dump. > > - misbehaving memory reclaim > > - minority of issues and the oom report is usually > > insufficient to drill down to the root cause. If the > > problem is reproducible then collecting vmstat data > > can give a much better clue. > > - high number of slab reclaimable objects or free swap > > are good indicators. Shrinkers data could be > > potentially helpful in the slab case but I really have > > hard time to remember any such situation. > > On non-production systems the situation is quite different. I can see > > how it could be very beneficial to add a very specific debugging data > > for subsystem/shrinker which is developed and could cause the OOM. For > > that purpose the proposed scheme is rather inflexible AFAICS. > > Considering that you're an MM guy, and that shrinkers are pretty much > universally used by _filesystem_ people - I'm not sure your experience > is the most relevant here? I really do not understand where you have concluded that. In those years of analysis I was not debugging my _own_ code. I was dealing with customer reports and I would not really blame them to specifically trigger any class of OOM reports. > The general attitude I've been seeing in this thread has been one of > dismissiveness towards filesystem people. Roman too; back when he was > working on his shrinker debug feature I reached out to him, explained > that I was working on my own, and asked about collaborating - got > crickets in response... This is completely off and it makes me _really_ think whether discussions like this on is really worth time. You have been presented arguments, you seem to be convinced that every disagreement is against you. Not the first time this is happening. Stop it please! As a matter of fact, you are proposeing a very specific form of debugging without showing that this is generally useful thing to do or even giving us couple of examples where that was useful in a production environment. This is where you should have started at and then we could help out to form an acceptable solution. Throwing "this does what we need, take it or leave" attitude is usualy not the best way to get your work merged. > Hmm.. > > Besides that, I haven't seen anything what-so-ever out of you guys to > make our lives easier, regarding OOM debugging, nor do you guys even > seem interested in the needs and perspectives of the filesytem people. > Roman, your feature didn't help one bit for OOM debuging - didn't even > come with documentation or hints as to what it's for. > > BPF? Please. > > Anyways. > > Regarding log spam: that's something this patchset already starts to > address. I don't think we needed to be dumping every single slab in the > system, for ~2 pages worth of logs; hence this patchset changes that to > just print the top 10. Increasing the threshold for slabs to be printed is something I wouldn't mind at all. > The same approach is taken with shrinkers: more targeted, less spammy > output. > > So now that that concern has been addressed, perhaps some actual meat: > > For one, the patchset adds tracking for when a shrinker was last asked > to free something, vs. when it was actually freed. So right there, we > can finally see at a glance when a shrinker has gotten stuck and which > one. The primary problem I have with this is how to decide whether to dump shrinker data and/or which shrinkers to mention. How do you know that it is the specific shrinker which has contributed to the OOM state? Printing that data unconditionally will very likely be just additional balast in most production situations. Sure if you are doing a filesystem development and you are tuning your specific shrinker then this might be a really important information to have. But then it is a debugging devel tool rather than something we want or need to have in a generic oom report. All that being said, I am with you on the fact that the oom report in its current form could see improvements. But please when adding more information please always focus on general usefulness. We have a very rich tracing capabilities which could be used for ad-hoc or very specific purposes as it is much more flexible. -- Michal Hocko SUSE Labs