From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E973C4167B for ; Thu, 30 Nov 2023 03:42:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C119E6B0434; Wed, 29 Nov 2023 22:42:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BC2B36B0435; Wed, 29 Nov 2023 22:42:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A62536B0436; Wed, 29 Nov 2023 22:42:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9756A6B0434 for ; Wed, 29 Nov 2023 22:42:56 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 60A521602F0 for ; Thu, 30 Nov 2023 03:42:56 +0000 (UTC) X-FDA: 81513224352.09.46C98AA Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf28.hostedemail.com (Postfix) with ESMTP id 82C2FC000F for ; Thu, 30 Nov 2023 03:42:53 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=fM87r5tK; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701315774; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rha3cUxUxmHIZK8eWzBvLAMSEva4ikfr6lW4zTpcG+g=; b=OsUTIckKxjUVNWigIfRRSwWs9FCwIkQ04uiKszUyZyx4chjn7iayf+SiHYqW8hjBmubp08 ILeRJ8SLF5PeLrYNh4qQAIZuEHS6DoE5PjBmdO/jSbDOhManrbXTLD5pmX/jJiew5f/R2A mHaJq7SO1mOKvNyuuR4UAG2OWRaTo/w= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=fM87r5tK; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701315774; a=rsa-sha256; cv=none; b=ibt4f51AR8DC1dU2ZoR8LD46dJUdSw8zoXsFrixuB5qBWN7S2NIEPPWhT6fq7VU4yFAC1a q7ReHQchbGm8HSq2Hzyig5dcFaHaGXBp8bOxLTUfvpXpCAejz+Q00fq+V12yxGp8jOiG/0 TP/E2/t41g573PbYOKFbBvl/Fm1ANxg= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1cfa7be5650so1426695ad.0 for ; Wed, 29 Nov 2023 19:42:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1701315772; x=1701920572; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=rha3cUxUxmHIZK8eWzBvLAMSEva4ikfr6lW4zTpcG+g=; b=fM87r5tKbJohpOuyCk1N6jbxYuu6TQVTdeqRk50DeGRlvZ+4L3x3T2zqNnQzHx/9YP NnjZeqrMcArEODKa4tEUmKl6+otrO/WDpZvxnIrghNMSRdvfTGrhkd5MM36W5LjPfXwz iTBsCeegWfrM4Lf86TUoX7cCT7yfrPupSIPUtqKfOjAo2EOKqsGdxASnKFgcZt6II7gt +qVxDthvMssk+zcLpJtzFQ1nlgWwG5KHUJ91GL/lzrWoqKG5ORjrRsLQf9xel8Yp4dpQ GBJQ9ulzB2dYmjOIw9CoPIRHCMsORAYnk7s7hZPQXR1hexEb489pTgMWExWp5gBsJGWg ojpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701315772; x=1701920572; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rha3cUxUxmHIZK8eWzBvLAMSEva4ikfr6lW4zTpcG+g=; b=DAJzMvJc/IJ6E+LBajMMBEHkGWddP6tw71/P+hyHDc/kaoVocopoXI5C2v9XLsPK8q V0ewLgo8TQ+6vQnnuRsVqDuLvBFWrk87CRfrAeBw6l5t7jh7Ba2eNXw+U85+4W8a0t4Z HR5pkqhAQj8AeyQuy9dhgt5zxQArppTLdfGgl5d8cGi/02zzmlh3YTFgDEm3vpr7X1qs FRv3LRqMtTRFxjeCMScrs2gP8Gsn71alkPQvKrTob0zNeUCBX/Ncdylqa/uNPuMW4c02 w2ty9JY9zu/1ssx1UDRLnOfzWGrvNaGnckXubFDJZLm2g2SNptEItUQ+tQ1nVuNSuMHS 7RAg== X-Gm-Message-State: AOJu0YzrQ6AruMcXqQeHgSCKvCCtf8/KE8vCel4tKrWr7gaOL7KX06YZ QCwKSN8OKKYRscDkEdxUxrDPaw== X-Google-Smtp-Source: AGHT+IEv74ESNDydAHvTgqgigBO6aX6962WeEbxyfe1Wxf4/rxmgXF1G2yNaJ/DmORsePZ3DvXdgvQ== X-Received: by 2002:a17:902:d4cb:b0:1cf:70a2:c26d with SMTP id o11-20020a170902d4cb00b001cf70a2c26dmr24549436plg.5.1701315771989; Wed, 29 Nov 2023 19:42:51 -0800 (PST) Received: from [10.84.152.29] ([203.208.167.146]) by smtp.gmail.com with ESMTPSA id y8-20020a17090322c800b001bc6e6069a6sm146856plg.122.2023.11.29.19.42.48 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Nov 2023 19:42:51 -0800 (PST) Message-ID: <6f56c8f4-77e3-4ad7-a5f8-a6235b047137@bytedance.com> Date: Thu, 30 Nov 2023 11:42:45 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 2/7] mm: shrinker: Add a .to_text() method for shrinkers Content-Language: en-US To: Kent Overstreet Cc: Michal Hocko , Roman Gushchin , Muchun Song , Linux-MM , linux-kernel@vger.kernel.org, Andrew Morton , Dave Chinner References: <20231123212411.s6r5ekvkklvhwfra@moria.home.lan> <4caadff7-1df0-45cc-9d43-e616f9e4ddb3@bytedance.com> <20231125003009.tbaxuquny43uwei3@moria.home.lan> <76A1EE85-B62C-49B3-889C-80F9A2A88040@linux.dev> <20231128035345.5c7yc7jnautjpfoc@moria.home.lan> <20231129231147.7msiocerq7phxnyu@moria.home.lan> <04f63966-af72-43ef-a65c-ff927064a3e4@bytedance.com> <20231130032149.ynap4ai47dj62fy3@moria.home.lan> From: Qi Zheng In-Reply-To: <20231130032149.ynap4ai47dj62fy3@moria.home.lan> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 82C2FC000F X-Rspam-User: X-Stat-Signature: baij5wi3665p7d36cow15uumpn8n6st7 X-Rspamd-Server: rspam01 X-HE-Tag: 1701315773-405921 X-HE-Meta: U2FsdGVkX18y1cNRSHiOrd2ZL6Yxu23jNTwa/po2KANAp32x7fgTBQ3ZQFL7sRVLxe5kSCVfaefNmVy0N3xFrPaCBl82iqc5eANe7YP7dhPqyzYMdfO+eZxz+yTFFMsOtt34M1BGh95IRwuFEkkxe+h4DznpLjimWp1+XvLE7iAs4/NoCtchrs0Sz4V5vexMzEcnpE55AkzF2zrgkTBHoSOmKhGbnUyqayvgXHRrQ06fmxEGuHD5vCBjn/xSwCFLUALOgbjfviPNGLo0YzysGfb3VqwUqGr88tTcl7tS6NuvpDVwD6iNGJzBsjNCUlkSZj/g+xQaooH87PEZv5zkRLOcGIuQalZ/brhV5XJAQCCGHNEVycfATsnTjA4Dnu1WcS2lP/OmEU1TCGMLqMmqHkfXJUt7uo/nnMX8fJ9wUGws2fF/K57EazxagMRPzieDaENieR1SddnTFeyPg7B14QRqORPBv+xCxQJMH0yER2emZxj9euTwyAmZrH+jV64LDxldH3YI/cv6zuHz4hXsh7OK6K2TvwQPXErZspeINImW+o/UdX4BwwLF1r/atcM5ia6CqhrM+ViT02sTkU4/EljS9UsHWjemlcUj+waTJXXmRdLmhkfmxYbGJBZeMd2k5F9e72US5VOwqrGGuwkWzwH4B6iwwx1rXkehUpJFIbT5ANaIz/s7a39RZqPhZfstbVbRb4DuRpjMthxLLrT93Lo8h3KyGA7WUzfWX2utcLnK4uUakWVihWF+yL7/HZq+HqAVPXCaPBtfJ8lHB/N9vyXUxAqFjqPfHdtKk94MGrlC2B5aACl1ncvVy1ydGMLu/MdTIAH7CzpAQG9dh/3ohJuCaxh77lR1UNWrMzdS9vBDw0jCLxGgKoRXqM3q67K33YwY8nfS6G8vVFzPARpZLqO+X8B0iLgAnveoZoEut63IQu71PdPVS83YF3362f+U3z83IgabpAIFQU0KEUv azKI2mYb SHlHjqbX1LYwMzpQ+5KltrlAfeu1iG96bQnxkHtPMHq8WXBlBHDaFPZXXjOCjhTsVFnWfMbBKSAldNSfyMurIQKBN61kex/AGuAVCQHF6LdPBLE3BoVifgA0FD7uRsn20ThjYiWACUkiseSIK/WjZUG4tgA6yWMCSVMGdbSH/4tMeJ3AzOMDPfceFBvxrkUZESMoZmhrhxK/EMQ393UQhGMjR2LsG+5SlcQ9DXWkJIsz7RAsEouMEnwg/PFfDY8AjHFXAdUKmjb8QMwxTg3D+1ya7eQxWqer1jzC2khq4je8wtfArkEM/EvSKvgRYmQUCVfpxWADFejJmeAzqlIUdf3+WSlvVI/6OVmH2FebNAr1QmEXEGMGWCEBCX/yk8EGhsbDZ8EBO90pmnoV9/hNAa6kJkbk5XXRXGUEwGx7j/L5OhLnnG4SbpjgYmkv5ZYE6LQtX0mbaHlRUTi012/p+VQFhS6tNM8Y1GxOVjOvAnHzIEjq7Z3qpD54JvlbhOAqkLlj4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2023/11/30 11:21, Kent Overstreet wrote: > On Thu, Nov 30, 2023 at 11:09:42AM +0800, Qi Zheng wrote: >> >> >> On 2023/11/30 07:11, Kent Overstreet wrote: >>> On Wed, Nov 29, 2023 at 10:14:54AM +0100, Michal Hocko wrote: >>>> On Tue 28-11-23 16:34:35, Roman Gushchin wrote: >>>>> On Tue, Nov 28, 2023 at 02:23:36PM +0800, Qi Zheng wrote: >>>> [...] >>>>>> Now I think adding this method might not be a good idea. If we allow >>>>>> shrinkers to report thier own private information, OOM logs may become >>>>>> cluttered. Most people only care about some general information when >>>>>> troubleshooting OOM problem, but not the private information of a >>>>>> shrinker. >>>>> >>>>> I agree with that. >>>>> >>>>> It seems that the feature is mostly useful for kernel developers and it's easily >>>>> achievable by attaching a bpf program to the oom handler. If it requires a bit >>>>> of work on the bpf side, we can do that instead, but probably not. And this >>>>> solution can potentially provide way more information in a more flexible way. >>>>> >>>>> So I'm not convinced it's a good idea to make the generic oom handling code >>>>> more complicated and fragile for everybody, as well as making oom reports differ >>>>> more between kernel versions and configurations. >>>> >>>> Completely agreed! From my many years of experience of oom reports >>>> analysing from production systems I would conclude the following categories >>>> - clear runaways (and/or memory leaks) >>>> - userspace consumers - either shmem or anonymous memory >>>> predominantly consumes the memory, swap is either depleted >>>> or not configured. >>>> OOM report is usually useful to pinpoint those as we >>>> have required counters available >>>> - kernel memory consumers - if we are lucky they are >>>> using slab allocator and unreclaimable slab is a huge >>>> part of the memory consumption. If this is a page >>>> allocator user the oom repport only helps to deduce >>>> the fact by looking at how much user + slab + page >>>> table etc. form. But identifying the root cause is >>>> close to impossible without something like page_owner >>>> or a crash dump. >>>> - misbehaving memory reclaim >>>> - minority of issues and the oom report is usually >>>> insufficient to drill down to the root cause. If the >>>> problem is reproducible then collecting vmstat data >>>> can give a much better clue. >>>> - high number of slab reclaimable objects or free swap >>>> are good indicators. Shrinkers data could be >>>> potentially helpful in the slab case but I really have >>>> hard time to remember any such situation. >>>> On non-production systems the situation is quite different. I can see >>>> how it could be very beneficial to add a very specific debugging data >>>> for subsystem/shrinker which is developed and could cause the OOM. For >>>> that purpose the proposed scheme is rather inflexible AFAICS. >>> >>> Considering that you're an MM guy, and that shrinkers are pretty much >>> universally used by _filesystem_ people - I'm not sure your experience >>> is the most relevant here? >>> >>> The general attitude I've been seeing in this thread has been one of >>> dismissiveness towards filesystem people. Roman too; back when he was >> >> Oh, please don't say that, it seems like you are the only one causing >> the fight. We deeply respect the opinions of file system developers, so >> I invited Dave to this thread from the beginning. And you didn’t CC >> linux-fsdevel@vger.kernel.org yourself. >> >>> working on his shrinker debug feature I reached out to him, explained >>> that I was working on my own, and asked about collaborating - got >>> crickets in response... >>> >>> Hmm.. >>> >>> Besides that, I haven't seen anything what-so-ever out of you guys to >>> make our lives easier, regarding OOM debugging, nor do you guys even >>> seem interested in the needs and perspectives of the filesytem people. >>> Roman, your feature didn't help one bit for OOM debuging - didn't even >>> come with documentation or hints as to what it's for. >>> >>> BPF? Please. >> >> (Disclaimer, no intention to start a fight, here are some objective >> views.) >> >> Why not? In addition to printk, there are many good debugging tools >> worth trying, such as BPF related tools, drgn, etc. >> >> For non-bcachefs developers, who knows what those statistics mean? >> >> You can use BPF or drgn to traverse in advance to get the address of the >> bcachefs shrinker structure, and then during OOM, find the bcachefs >> private structure through the shrinker->private_data member, and then >> dump the bcachefs private data. Is there any problem with this? > > No, BPF is not an excuse for improving our OOM/allocation failure > reports. BPF/tracing are secondary tools; whenever we're logging > information about a problem we should strive to log enough information > to debug the issue. > > We've got junk in there we don't need: as mentioned before, there's no > need to be dumping information on _every_ slab, we can pick the ones > using the most memory and show those. > > Similarly for shrinkers, we're not going to be printing all of them - > the patchset picks the top 10 by objects and prints those. Could > probably be ~4, there's fewer shrinkers than slabs; also if we can get > shrinkers to report on memory owned in bytes, that will help too with > deciding what information is pertinent. I'm not worried about the shrinker's general data. What I'm worried about is the shrinker's private data. Except for the corresponding developers, others don't know the meaning of the private statistical data, and we have no control over the printing quantity and form of the private data. This may indeed cause OOM log confusion and failure to automatically parse. For this, any thoughts? > > That's not a huge amount of information to be dumping, and to make it > easier to debug something that has historically been a major pain point. > > There's a lot more that could be done to make our OOM reports more > readable and useful to non-mm developers. Unfortunately, any time > changing the show_mem report the immediate reaction seems to be "but > that will break my log parsing/change what I'm used to!"...