From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB2ECC25B74 for ; Sun, 2 Jun 2024 20:39:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 267986B00BF; Sun, 2 Jun 2024 16:39:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 217076B00C0; Sun, 2 Jun 2024 16:39:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DE8A6B00C1; Sun, 2 Jun 2024 16:39:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E34456B00BF for ; Sun, 2 Jun 2024 16:38:59 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 329621A0329 for ; Sun, 2 Jun 2024 20:38:59 +0000 (UTC) X-FDA: 82187112798.30.3780D97 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by imf29.hostedemail.com (Postfix) with ESMTP id 6080A12000D for ; Sun, 2 Jun 2024 20:38:57 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fGi7E6Js; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of yuzhao@google.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717360737; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AfldORuTjlVpAnJ3Ucw4knFWPlGKfdDgSo69+8PCJyE=; b=ri/oF9pP9nAb7YvlM+R5cs4Ib/iaHTPTD5U2YUasOQzLmxrvFIotWGQlr73LXemeIK00jK X8ZcjpBm4BYYEb3C+UtGKwGpdSYLjcIb+aHb4ZuVXVBBptVKsnWYze45BeoQ0dj4za0BIa 6gmSTAqeYq0Z6ivkldGrS5Qrh9UdSy0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717360737; a=rsa-sha256; cv=none; b=fW08yQUWZwtLblfo03VLoFijITbe5NH+PpCojZGNb2e/YIfN4mQAq7acZjIF8XoHhb8qYq Haiw40YDVPNoWT4NmULfZqsSOdRqohGWlqg/s133gfa0cijcAEO7RCLo62MHiDbl/QShzC 2j/qkUOne9QrPff5VT+nsNzDH+Iksac= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fGi7E6Js; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of yuzhao@google.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=yuzhao@google.com Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-42133fbe137so53015e9.0 for ; Sun, 02 Jun 2024 13:38:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717360736; x=1717965536; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=AfldORuTjlVpAnJ3Ucw4knFWPlGKfdDgSo69+8PCJyE=; b=fGi7E6JsCKIom40Sqw5QgP3T7G2/K4Pb3Huz39YLz8DiSCo4pkLIi4OlalW1AXiZF+ u3f7ZDAeBThtucqdAhF1ejcFkRiwoYqtU580Ol8eARGlQAnQodOGHP06khnhhCX1MUO9 sdY2+7SLrEuQ/i23+AdFIX70K3yA4+jx7Gu9bfmjUdG3MRD2IvMPvwlQ6Z+khT2ejMYs iSBt8fERrts9JbBqMkS1esYRuaLRvyNODuKNgzpt5y+BXBuK+fYF7+/m9u7ALxOELAcI QLFm0Pdg6suyD6fvdPu1uz8sV7eEXusGcUWf10In6AwYkSaDNDgd8jpalmqm927Jl3F8 4LDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717360736; x=1717965536; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AfldORuTjlVpAnJ3Ucw4knFWPlGKfdDgSo69+8PCJyE=; b=dxUWmXsHxSb3VYj+Iv6Wndfjcl/Emz8qkviRS8DOb45ZHMxaNgNLasZZNJsJKRmQ2r qv4y0fWpqg2LGqYBR3eDQ6SD6mxbTJtSlOkmaD5FgNb9b1rWG0+/pG9NnkCKsprq8JRF qI2wsOJ841xoxK4e++4v3EJClLED1HokPZDNuLd/M3HN/n/Z/8MFGBzBMYBxBTVcxXyb lYTXLWYiloDYVb/suGvwxNmqjpdYVIgHisYFUeA5zXEY8Q/293Bd0E1vYSa5dqJw3Nzb lExDa9gSSnIlxnx0X8XnfwxcW7N/mffI6Iny668l3s5rMWhxYJHuWA4T/33RoCNK64RI 2dvg== X-Gm-Message-State: AOJu0Ywo1e0mABs9DXsuUieuTpuSYXaVcvZkN/XOXwryHqzXzV/O1xt3 YGN+d2vLFTXkhlSwFOMFJShU6SRqDkdJcWhT8P81Wofk6P80Q43sRHbdFN/X1AWuiWz5Lp3qlNc UPAbMOTnc3FvjufKBItNDXyCX1yfenEQuWcHb X-Google-Smtp-Source: AGHT+IFFroboW96WPdvdwnL4DyxgbhQ5ulmPuS+yGKYAppow/rIDCpVjA8jHAiaBb0HM8OYqH+auJiagPkOKXUL7hhs= X-Received: by 2002:a05:600c:1d27:b0:421:328e:99db with SMTP id 5b1f17b1804b1-421358b2ea5mr2182435e9.1.1717360735443; Sun, 02 Jun 2024 13:38:55 -0700 (PDT) MIME-Version: 1.0 References: <20240508202111.768b7a4d@yea> <20240515224524.1c8befbe@yea> <20240602200332.3e531ff1@yea> In-Reply-To: <20240602200332.3e531ff1@yea> From: Yu Zhao Date: Sun, 2 Jun 2024 14:38:18 -0600 Message-ID: Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc) To: Erhard Furtner Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: 63h6khwwumys7ij7in5t4dw5j6rzfp8a X-Rspamd-Queue-Id: 6080A12000D X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1717360737-48200 X-HE-Meta: U2FsdGVkX1/PevR2vxGcDRbzMeBleXvM1XeZQ91KVG9lTO8Fq7Qz/ONAI79dnYrfObsrJQ5MxVwaR0Nr/kRkDw89ltnyRmC/tlvAWXdMzZiV8hOdp2930caDMTvcwsga4DTCwzgWcpue70SAuwnevWu1e1YPI6RKtqQlP3EVkFnl0YtcPejzH7Z0loXAr8A6iJR0jDgSWsHEvPIUOAAUKrf9wjPacOn4xZGGsmSW8cejUAGcA19F3PDB/Nwd/M4tV1ZQGNCpjnaZ4+pP1qTCg/XAIOdgSKIn7qe/CLCiPLh8f9I5nrAxs7GUQGa9MgiuuwBrO4b7WSxWCkSSHDW3rqvFQ3hrkOKlBZxx6utGOPFkjax6fUs/juiv8ONCenuZwcv1GBhPmd7X/uxmAFibb+bzELVFhkk5Bnqfy/9GRG7ABQZX92awz/4x5i15G5/9/sev8HQmqpC8fonSaIAhy0zA+av8xE5ZjH4bmCQYl2vpCtYhD9yZATUSO7FP4Br4V2Api7o61tTW/hxVHvzbXmPTjN/zpaElWnMRmQdAOlBYbHUX0sOWBZldLB2YAHcFAoLUo0AlnFif2tP7x4Zaf+Z21kqZ1xtaflAy6YCsjz8tU4mlrbDnX4kzOA2x9PHyvzclCu9/XbdJElD7zX0fQjNCZ3nhT5asy9yuaTJNXpweoCvH6+5XuEfQh/16UOMsCl+aeN3cYEtfbREW1Qrk62rl2E8cay60ffxwrjomD2soygftKH+0AVcsqQ4x37GH5FtC/fTCpc8u/AHO4n8pRyG4/7rQOJ327lHydRmKWub0KcFjMzBGGvSZBgJNyDxH0Ab214jIGJ42gsz84SwkCyMPebVnQBjJVUjVziQ3LXlp25LilHj7jQRLAkg+ErvPSeFljLuaPUk3NCwFkhC8TljFNmu5NTmFBt7KqyRllPixmEjxumgtRqfd5ipW2H/z690rwAWpz70uVTRVSvK fhyCN0a8 mgF9TW1qwPKzM+M+T6rg1XQ2Qb9gSBwUVzktQj4QifOJMXqJxAupQfFsZl7tx52S3Vr+Vvt+9JSxL47+OJn0ozaVsP/ydXcbGz+Vwv09woaVri3qe//jVNNwxo2DwaRA3SsEauCm2P1U255oQvacm+PJXXAz6bs6+ZEadi92kLZsMRUU0ORLphC8zy6/Cgbhwhaj0g0e5SuDXknyt/9+6OIwq2tEAJZyUM762AoQcDOu6fox+Vk0FMaEjoM23al48hhrWXPtkCB3QtnzfcdhmB4mebPv+xHZYze/Hp0Tz8F1wwgDmhwUHuRX0jiFZrTwXrFnrL3+NYqLxWYYtNN23ECqAa5iwS9anXzcyj4gGwaan74KM6zt8BI2Mbv4iXYqFUVBSH0KLsipAVIMI4b3d/F3cJXaFvhswHU4d X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, Jun 2, 2024 at 12:03=E2=80=AFPM Erhard Furtner wrote: > > On Sat, 1 Jun 2024 00:01:48 -0600 > Yu Zhao wrote: > > > Hi Erhard, > > > > The OOM kills on both kernel versions seem to be reasonable to me. > > > > Your system has 2GB memory and it uses zswap with zsmalloc (which is > > good since it can allocate from the highmem zone) and zstd/lzo (which > > doesn't matter much). Somehow -- I couldn't figure out why -- it > > splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone: > > > > [ 0.000000] Zone ranges: > > [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] > > [ 0.000000] Normal empty > > [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff] > > > > The kernel can't allocate from the highmem zone -- only userspace and > > zsmalloc can. OOM kills were due to the low memory conditions in the > > DMA zone where the kernel itself failed to allocate from. > > > > Do you know a kernel version that doesn't have OOM kills while running > > the same workload? If so, could you send that .config to me? If not, > > could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out > > of ideas at the moment.) > > > > Thanks! > > Hi Yu! > > Thanks for looking into this. > > The reason for this 0.25GB DMA / 1.75GB highmem split is beyond my knowle= dge. I can only tell this much that it's like this at least since kernel v4= .14.x (dmesg of an old bugreport of mine at https://bugzilla.kernel.org/sho= w_bug.cgi?id=3D201723), I guess earlier kernel versions too. > > Without CONFIG_HIGHMEM the memory layout looks like this: > > Total memory =3D 768MB; using 2048kB for hash table > [...] > Top of RAM: 0x30000000, Total RAM: 0x30000000 > Memory hole size: 0MB > Zone ranges: > DMA [mem 0x0000000000000000-0x000000002fffffff] > Normal empty > Movable zone start for each node > Early memory node ranges > node 0: [mem 0x0000000000000000-0x000000002fffffff] > Initmem setup node 0 [mem 0x0000000000000000-0x000000002fffffff] > percpu: Embedded 29 pages/cpu s28448 r8192 d82144 u118784 > pcpu-alloc: s28448 r8192 d82144 u118784 alloc=3D29*4096 > pcpu-alloc: [0] 0 [0] 1 > Kernel command line: ro root=3D/dev/sda5 slub_debug=3DFZP page_poison=3D1= netconsole=3D6666@192.168.2.8/eth0,6666@192.168.2.3/A8:A1:59:16:4F:EA debu= g > Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear) > Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear) > Built 1 zonelists, mobility grouping on. Total pages: 194880 > mem auto-init: stack:all(pattern), heap alloc:off, heap free:off > Kernel virtual memory layout: > * 0xffbdf000..0xfffff000 : fixmap > * 0xff8f4000..0xffbdf000 : early ioremap > * 0xf1000000..0xff8f4000 : vmalloc & ioremap > * 0xb0000000..0xc0000000 : modules > Memory: 761868K/786432K available (7760K kernel code, 524K rwdata, 4528K = rodata, 1100K init, 253K bss, 24564K reserved, 0K cma-reserved) > [...] > > With only 768 MB RAM and 2048K hashtable I get pretty much the same "kswa= pd0: page allocation failure: order:0, mode:0xcc0(GFP_KERNEL),nodemask=3D(n= ull),cpuset=3D/,mems_allowed=3D0" as with the HIGHMEM enabled kernel at > running "stress-ng --vm 2 --vm-bytes 1930M --verify -v". > > I tried the workload on v6.6.32 LTS where the issue shows up too. But v6.= 1.92 LTS seems ok! Triple checked v6.1.92 to be sure. > > Attached please find kernel v6.9.3 dmesg (without HIGHMEM) and kernel v6.= 1.92 .config. Thanks. I compared the .config between v6.8.9 (you attached previously) and v6.1.92 -- I didn't see any major differences (both have ZONE_DMA, HIGHMEM, MGLRU and zswap/zsmalloc). Either there is something broken between v6.1.92 and v6.6.32 (as you mentioned above), or it's just a kernel allocation bloat which puts the DMA zone (0.25GB) under too heavy pressure. The latter isn't uncommon when upgrading to a newer version of the kernel. Could you please attach the dmesg from v6.1.92? I want to compare the dmegs between the two kernel versions as well -- that might provide some hints.