From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C1BDC433F5 for ; Thu, 21 Apr 2022 18:42:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B069B6B0072; Thu, 21 Apr 2022 14:42:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AB4F96B0073; Thu, 21 Apr 2022 14:42:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92F576B0074; Thu, 21 Apr 2022 14:42:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 821D36B0072 for ; Thu, 21 Apr 2022 14:42:17 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 578382789F for ; Thu, 21 Apr 2022 18:42:17 +0000 (UTC) X-FDA: 79381756314.26.315FAC1 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf11.hostedemail.com (Postfix) with ESMTP id 0965040024 for ; Thu, 21 Apr 2022 18:42:14 +0000 (UTC) Received: by mail-qv1-f42.google.com with SMTP id ke5so2106461qvb.5 for ; Thu, 21 Apr 2022 11:42:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=8ILPYEi6EpMIBrPlORAyxArgIQC8qGPRB7ecYXue0XE=; b=qDmzbvel2H0KQCr02PQ03kcpCNG1E0o1whuTsEYnThkFJUJ3sNvJwvYgqQNIfs8nb2 +uxVbcWINFlGgkA0JTlJ17pQPcwYqapj7+NXT+Vl1uRZwntc/MjFx/jQNWeui09wDO6I enMYLsEQYMMBazGc2rwLdSoHwjEA+d2O/26j3XPoaF6ZxCaHekns4iN8ZA97dOxIl0AM lxo81oYseH4u3QOktWkxdxYIi4eyezIegBO5RzwJtz6gQ10uA+wEbGa0J0VIuNC55sJR 4wBB8ugemw65dSMTj2vi/ZTx44ijQ4ZpCRZSnKz+3QGyW1zQs7fdbetClU9+jUxoj4db 58MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=8ILPYEi6EpMIBrPlORAyxArgIQC8qGPRB7ecYXue0XE=; b=CfOhNe3o+t2Xo/73Osltn20p63piEkMGitxyGD8bBrw1zkZI0SCLtr/ZU669jLIZgD iuYDF3djFiArdVizzUrcKVhgVMNjvPuwTSnT/yBJxR8iVLiPTsedzC5ADjSRE2t/+SzT 63vf+PA22j5KB0mwZSK0/mbCcpzCCj8bj26BGPMjwdZLwe/vPIrOCLNe4zoD9IiIL9X6 cDi1MiLLOtP1GmDDP4Z8FiQRmzOiAQiDYUyGnHsiakPj4MTPUYZOlQqOgLGOVMVkHFNe 6OUgvkVDMNM0qkHUR7Sr/VRBo68+ImunScrz1jyjHkUYjiEFqiUJOs5NF+HBc8wpG/Nm 2dpQ== X-Gm-Message-State: AOAM5308mqvPDy5Yne08WHu55LlCLHs0UWkXzUlicAKQHN43UdRTYZsh 1fEOjDe6BH8nC21sLf+/KA== X-Google-Smtp-Source: ABdhPJzQJ3l45+fADcNjJxP40bzPfKP0UgQ3xjcQbrwZ3ZPLYX0JvwMaErQtwe4c7GeKvQgDEaE8Yw== X-Received: by 2002:a05:6214:1ccd:b0:443:652e:69d with SMTP id g13-20020a0562141ccd00b00443652e069dmr681924qvd.114.1650566536135; Thu, 21 Apr 2022 11:42:16 -0700 (PDT) Received: from moria.home.lan (c-73-219-103-14.hsd1.vt.comcast.net. [73.219.103.14]) by smtp.gmail.com with ESMTPSA id 9-20020a05620a070900b0069e60da45aasm3171683qkc.60.2022.04.21.11.42.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Apr 2022 11:42:15 -0700 (PDT) Date: Thu, 21 Apr 2022 14:42:13 -0400 From: Kent Overstreet To: Michal Hocko Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, roman.gushchin@linux.dev, hannes@cmpxchg.org Subject: Re: [PATCH 3/4] mm: Centralize & improve oom reporting in show_mem.c Message-ID: <20220421184213.tbglkeze22xrcmlq@moria.home.lan> References: <20220419203202.2670193-1-kent.overstreet@gmail.com> <20220419203202.2670193-4-kent.overstreet@gmail.com> <20220420165805.lg4k2iipnpyt4nuu@moria.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0965040024 X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=qDmzbvel; spf=pass (imf11.hostedemail.com: domain of kent.overstreet@gmail.com designates 209.85.219.42 as permitted sender) smtp.mailfrom=kent.overstreet@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: qxemd8fd6gjxozoihah9hgbsphgotnme X-HE-Tag: 1650566534-253133 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000035, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Apr 21, 2022 at 11:18:20AM +0200, Michal Hocko wrote: > On Wed 20-04-22 12:58:05, Kent Overstreet wrote: > > On Wed, Apr 20, 2022 at 08:58:36AM +0200, Michal Hocko wrote: > > > On Tue 19-04-22 16:32:01, Kent Overstreet wrote: > > > > This patch: > > > > - Moves lib/show_mem.c to mm/show_mem.c > > > > > > Sure, why not. Should be a separate patch. > > > > > > > - Changes show_mem() to always report on slab usage > > > > - Instead of reporting on all slabs, we only report on top 10 slabs, > > > > and in sorted order > > > > - Also reports on shrinkers, with the new shrinkers_to_text(). > > > > > > Why do we need/want this? It would be also great to provide an example > > > of why the new output is better (in which cases) than the existing one. > > > > Did you read the cover letter to the patch series? > > Nope, only this one made it into my inbox based on my filters. I usually > try to fish out other parts of the thread but I didn't this time. > Besides it is always better to have a full patch description explain not > only what has been changed but why as well. > > > But sure, I can give you an example of the new output: > > Calling out the changes would be really helpful, but I guess the crux > is here. > > > 00177 16644 pages reserved > > 00177 Unreclaimable slab info: > > 00177 9p-fcall-cache total: 8.25 MiB active: 8.25 MiB > > 00177 kernfs_node_cache total: 2.15 MiB active: 2.15 MiB > > 00177 kmalloc-64 total: 2.08 MiB active: 2.07 MiB > > 00177 task_struct total: 1.95 MiB active: 1.95 MiB > > 00177 kmalloc-4k total: 1.50 MiB active: 1.50 MiB > > 00177 signal_cache total: 1.34 MiB active: 1.34 MiB > > 00177 kmalloc-2k total: 1.16 MiB active: 1.16 MiB > > 00177 bch_inode_info total: 1.02 MiB active: 922 KiB > > 00177 perf_event total: 1.02 MiB active: 1.02 MiB > > 00177 biovec-max total: 992 KiB active: 960 KiB > > 00177 Shrinkers: > > 00177 super_cache_scan: objects: 127 > > 00177 super_cache_scan: objects: 106 > > 00177 jbd2_journal_shrink_scan: objects: 32 > > 00177 ext4_es_scan: objects: 32 > > 00177 bch2_btree_cache_scan: objects: 8 > > 00177 nr nodes: 24 > > 00177 nr dirty: 0 > > 00177 cannibalize lock: 0000000000000000 > > 00177 > > 00177 super_cache_scan: objects: 8 > > 00177 super_cache_scan: objects: 1 > > How does this help to analyze this allocation failure? You asked for an example of the output, which was an entirely reasonable request. Shrinkers weren't responsible for this OOM, so it doesn't help here - are you asking me to explain why shrinkers are relevant to OOMs and memory reclaim...? Since shrinkers own and, critically, _are responsible for freeing memory_, a shrinker not giving up memory when asked (or perhaps not being asked to give up memory) can cause an OOM. A starting point - not an end - if we want to improve OOM debugging is at least being able to see how much memory each shrinker owns. Since we don't even have that, number of objects will have to do. The reason for adding the .to_text() callback is that shrinkers have internal state that affects whether they are able to give up objects when asked - the primary example being object dirtyness. If your system is using a ton of memory caching inodes, and something's wedged writeback, and they're nearly all dirty - you're going to have a bad day. The bcachefs btree node node shrinker included as an example of what we can do with this: internally we may have to allocate new btree nodes by reclaiming from our own cache, and we have a lock to prevent multiple threads from doing this at the same time, and this lock also blocks the shrinker from freeing more memory until we're done. In filesystem land, debugging memory reclaim issues is a rather painful topic that often comes up, this is a starting point...