From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1144C25B78 for ; Tue, 4 Jun 2024 20:52:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED4E96B007B; Tue, 4 Jun 2024 16:52:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E5E196B0082; Tue, 4 Jun 2024 16:52:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD7F86B0083; Tue, 4 Jun 2024 16:52:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id AC76D6B007B for ; Tue, 4 Jun 2024 16:52:17 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3A5C01A0B32 for ; Tue, 4 Jun 2024 20:52:17 +0000 (UTC) X-FDA: 82194403914.14.87B1078 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf22.hostedemail.com (Postfix) with ESMTP id 735B1C0010 for ; Tue, 4 Jun 2024 20:52:15 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=BGnOnSbd; spf=pass (imf22.hostedemail.com: domain of vbabka@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=vbabka@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717534335; a=rsa-sha256; cv=none; b=CoQTjNhXkhWsxMAcOeXiaAUjAQi2ayQSI/cIeyjyWFgYEVqz9MvErQos1kg2Tk3zIRw7fA cWE5g3XmYbFAxJhkv6etDxF/6UG8ICp4zGUsrIT4uPhFgkdqH7RcpoRH91Xp933ZPP7jsj w6iA9xkn6Jk4LF/NPAWjBjVaAg/57dE= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=BGnOnSbd; spf=pass (imf22.hostedemail.com: domain of vbabka@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=vbabka@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717534335; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=amQzqMiBqhOdkRN/9wi6uWfvhqSAJF4HagIHhNACgsY=; b=8NCRPQmDX2XtOP4d1oyl7PGTHkva4IpnVqUMZBqE9TaW9CdAJ9nwAlZnsjRRqYDGsz7XaX hC/CPyZQLL3UtLxJ8oWd/BgAcXpGRLj/sAAMGLfZy9L1+09rZ+FkR4Gmz47/e9NVXdT/oM t9uQqwiKyAycL7Iz+mwkkAOzRo1OnOQ= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 7209F614F1; Tue, 4 Jun 2024 20:52:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E4A9DC2BBFC; Tue, 4 Jun 2024 20:52:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717534334; bh=y+8CKHrB4O4j80cxj7GlaoyAv7eWMbh8YsZykcrvLj4=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=BGnOnSbd85K0EPEFf3nTeVbhYOtDqa45A97NwY2zZBnWV4ojBrqYs0IcfHwTfsIOJ joJnGI1VJ/EwD0prkDWc/P43JZKVMOCCMVLTUuAdlqMVgxZbboaYnOcnFNoiZoW3J0 2xmB1ByBVgutfJsC8iD37OAus1bzrPOMab0j1ap61vCefTDHP7nfNGJa+WDPtO0Bgg AGYlYht2Ca5HdLUXLA8YXiNCGXU/9QjkE3K9dAwERt9nF4HyW+X7VPeV0kEsqsjnDW Bc2Ie90vXwIDb+F5KdGiR0IJezHMoHknk827GbopURf5wTW9VT0DvygiDOzzCdx1+S thIUZ5OAxOruw== Message-ID: <5c69cee6-f101-4c86-b38a-7c5c8679ea9b@kernel.org> Date: Tue, 4 Jun 2024 22:52:09 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc) To: Yosry Ahmed , Erhard Furtner Cc: Yu Zhao , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org References: <20240508202111.768b7a4d@yea> <20240515224524.1c8befbe@yea> <20240602200332.3e531ff1@yea> <20240604001304.5420284f@yea> Content-Language: en-US From: "Vlastimil Babka (SUSE)" In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 735B1C0010 X-Stat-Signature: piariz67scpp1a47xdrhbn3rfqgwui69 X-HE-Tag: 1717534335-720244 X-HE-Meta: U2FsdGVkX1+EihKUeQO/ekn/aRwH3fXGt4qLgV+yL1YQfi9Aw65BPTzFhzj+/GXzWBe+xYShPg2qtdax8WRmLqMjMMW5BfymwKsPXw7r2LmkcO1zUAYGumc9hnIyRDpCF3XKLOXrp90/JBtzSKMuVx9NJGOzpvdzgM088KsveRHuqTXIbcRWTUbXbyuvyDm7SusFQcZXp6LL1+VG5troGytijJOe4zxh6yaAihEGslW7hr/fzaU3Uf3B6J95RbRe0vthU19fbvVrVdQ9fthulo1hvpXzxWRtSpHmRxdUU60Rw401SVDlILAgEHU5066CAKe7RVz9+OKs+aAW/FmyMSNsauy1AAuznMXDpZKTLsc7/3tLaZ+zyeSrlDMX5Vmc2Cv9n6hlnQt/1L4LCyebBRnqPQBWsCQ0cNJNJG25+ZEZTRd7FXACeu8j16enbL+0LHLI9sh4If3kNDm/cn9K+U9GUWtxu0gDfyFxBbe4XUnyTGkHf/OCTRH8Kcf36GtQQmWwoghrQbktKZ/ycly7Cr/alQlrVy4ARa2xcEAE/N2HTXgl0uEvp5e8w3m6vT6RTMi8uqLJUM4BAeHIzr7BEJsQyQ2XSw1CoTLN+kVwKBBnCvYHDudcgI/FWopNkfCw1NaL3auCOV4S0W50iwiEexyK7p2rq0ZcjIV52PJEIj3vLgMRx/sfkSXSpO+/U6Ik+B36ZFeg6SKUacC1wW2/3tLdWDgV2cTc5otuGlsPd5YTaZ1dI8rQIwIVejhyH7SkCLzO/5qJCE0L0ON+24diuLX7sxLbM8dEuumHc7dv4+HlkGUtNTR79Ml64fi9m5qGUcjSoXInPPLi/QzXCV6ripOpV/QDkY6XACFQ4A/axefIp3foKdQ30Oh5Ihd4lMmbMHomeInNc4E9P3pyRTHv+x33y1ipPLCfNravxhg9P2BVJ01ZmanoTB3lbr1WEAOaT/hafq91Y+ymeKADAfh sxGtQc/g igBeQ6BLnm4Hl8rjPOWvApz95N77pIcqXe0SF1rrDZRWmZen0YtPlz8IXjcqH4pmrWY9x6rxy2wAyrEIiknG2D9NOhWsLbOuIwZG56yWktuR9WeklcjxsZSABl1L9+qA7Oi87w7hJ94wFBgXDz/a3q3y2yRk2ms+gIOxCknYsIcPahgXJatIOM3meBBSpzSpCGftFFtG6VNlXWyu+DZibAj92VzoDo2FNCB87V/4743+Sn+H0MoZ98LSU+ck33Gf1k5VyYWku29FshakbRqMCb5JewHptFeq/gUa2HLOqSSenKf3DHllR3wuTW+REuWkrrIjwmsy35wJv76yhZO6t8HXJ2O/bGygSx97YHXxsec7ybaYzXakgbfrdTvKMPjzhKtaB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 6/4/24 1:24 AM, Yosry Ahmed wrote: > On Mon, Jun 3, 2024 at 3:13 PM Erhard Furtner wrote: >> >> On Sun, 2 Jun 2024 20:03:32 +0200 >> Erhard Furtner wrote: >> >> > On Sat, 1 Jun 2024 00:01:48 -0600 >> > Yu Zhao wrote: >> > >> > > The OOM kills on both kernel versions seem to be reasonable to me. >> > > >> > > Your system has 2GB memory and it uses zswap with zsmalloc (which is >> > > good since it can allocate from the highmem zone) and zstd/lzo (which >> > > doesn't matter much). Somehow -- I couldn't figure out why -- it >> > > splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone: >> > > >> > > [ 0.000000] Zone ranges: >> > > [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] >> > > [ 0.000000] Normal empty >> > > [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff] >> > > >> > > The kernel can't allocate from the highmem zone -- only userspace and >> > > zsmalloc can. OOM kills were due to the low memory conditions in the >> > > DMA zone where the kernel itself failed to allocate from. >> > > >> > > Do you know a kernel version that doesn't have OOM kills while running >> > > the same workload? If so, could you send that .config to me? If not, >> > > could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out >> > > of ideas at the moment.) >> >> Ok, the bisect I did actually revealed something meaningful: >> >> # git bisect good >> b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit >> commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17 >> Author: Yosry Ahmed >> Date: Tue Jun 20 19:46:44 2023 +0000 >> >> mm: zswap: multiple zpools support > > Thanks for bisecting. Taking a look at the thread, it seems like you > have a very limited area of memory to allocate kernel memory from. One > possible reason why that commit can cause an issue is because we will > have multiple instances of the zsmalloc slab caches 'zspage' and > 'zs_handle', which may contribute to fragmentation in slab memory. > > Do you have /proc/slabinfo from a good and a bad run by any chance? > > Also, could you check if the attached patch helps? It makes sure that > even when we use multiple zsmalloc zpools, we will use a single slab > cache of each type. As for reducing slab fragmentation/footprint, I would also recommend these changes to .config: CONFIG_SLAB_MERGE_DEFAULT=y - this will unify the separate zpool caches as well (but the patch still makes sense), but also many others CONFIG_RANDOM_KMALLOC_CACHES=n - no 16 separate copies of kmalloc caches although the slabinfo output doesn't seem to show CONFIG_RANDOM_KMALLOC_CACHES in action, weirdly. It was enabled in the config attached to the first mail. Both these changes mean giving up some mitigation against potentai lvulnerabilities. But it's not perfect anyway and the memory seems really tight here.