From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 906E4C25B75 for ; Mon, 3 Jun 2024 22:13:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 251136B0082; Mon, 3 Jun 2024 18:13:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 200C56B0095; Mon, 3 Jun 2024 18:13:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EFFD6B0096; Mon, 3 Jun 2024 18:13:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E53446B0082 for ; Mon, 3 Jun 2024 18:13:16 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6C48714026B for ; Mon, 3 Jun 2024 22:13:16 +0000 (UTC) X-FDA: 82190979192.29.462288B Received: from mout-p-201.mailbox.org (mout-p-201.mailbox.org [80.241.56.171]) by imf23.hostedemail.com (Postfix) with ESMTP id 19883140002 for ; Mon, 3 Jun 2024 22:13:12 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=mailbox.org header.s=mail20150812 header.b=jMkzvXns; dmarc=pass (policy=reject) header.from=mailbox.org; spf=pass (imf23.hostedemail.com: domain of erhard_f@mailbox.org designates 80.241.56.171 as permitted sender) smtp.mailfrom=erhard_f@mailbox.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717452793; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A4mvgXvzsnDLCwao81wHmfd72l0a/3E0/ihIluzp53E=; b=kiMWrpj1I7zi/hLQbXUZ7P88S5TObTbJKccxJWwDX8e7jBBAYAKJYs8BRn5NHOAFGL/k07 4Vqw1OUhxXb7nsBuXSFeOz8aHjCTlziEuUmCNaVAXrEsWf3WD4FSf6EjIThmiQgXoq7mdk XuzoNFpfHuCd7OuFeDvuueaI7XHZIX0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=mailbox.org header.s=mail20150812 header.b=jMkzvXns; dmarc=pass (policy=reject) header.from=mailbox.org; spf=pass (imf23.hostedemail.com: domain of erhard_f@mailbox.org designates 80.241.56.171 as permitted sender) smtp.mailfrom=erhard_f@mailbox.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717452793; a=rsa-sha256; cv=none; b=BK08+hxoi4NsETWztLKxeB+gzMbo+bjMFHQ5+3IlHzk/m+KVZmAr5kdypeXSmM74x+G7hJ IDxsvR4AQJePlSCO5ronxOJPMlJ/SQG1RDWHLusQgTw9j81Z3gEN7SJbDqnk9TrV4QPa3N cznkg0KOBDIKG0Y8shmWcae8z2e+ZaU= Received: from smtp202.mailbox.org (smtp202.mailbox.org [IPv6:2001:67c:2050:b231:465::202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-201.mailbox.org (Postfix) with ESMTPS id 4VtSfC58Zlz9sZG; Tue, 4 Jun 2024 00:13:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mailbox.org; s=mail20150812; t=1717452787; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=A4mvgXvzsnDLCwao81wHmfd72l0a/3E0/ihIluzp53E=; b=jMkzvXnsDbJMZoaVvMV9HUytnV1y8iZ5KUo9y1GdlxePPX3XfQBk/s6pqxLBiUcOLlfkVk 4ayMfjjbjpLwwPRDTtLkMZ5InIqxthuD8ZRRSJim7C9L/McHiAtiPQYFKFgLEHGnmKr0jg lC2U+4tuwDf2dvtnn9jtqQ3sO8A+XweQPmmOJmBAphNUhej6GQiaB4ZsNl7D/rvOH+Q4bS Tn5EGAmlejUafEI1k+QohoXu2A/Qsz5asXCr/dzhX1DHEkx6xcjLedRWCTcJ+xtZHcIDjX ndxSvjfgjMTvemQUY67wki8rSZf9iWh39Q0vuis64IX65xKphxiJwA3SQFM/1w== Date: Tue, 4 Jun 2024 00:13:04 +0200 From: Erhard Furtner To: Yu Zhao Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Yosry Ahmed Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc) Message-ID: <20240604001304.5420284f@yea> In-Reply-To: <20240602200332.3e531ff1@yea> References: <20240508202111.768b7a4d@yea> <20240515224524.1c8befbe@yea> <20240602200332.3e531ff1@yea> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/L6u2AePrH2ig2h2isRGtuja" X-MBO-RS-ID: 1fbfa6395497c39efc6 X-MBO-RS-META: 46pgksj1cfgiiro3owzb3sxib5g9z6g6 X-Rspamd-Queue-Id: 19883140002 X-Stat-Signature: f6kwypbwew4i56f5m6bid3p7n5j3juyq X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1717452792-8661 X-HE-Meta: U2FsdGVkX1/1aGuMWBxA+ebIQF3/L1jN5YbTpiqWxC+EregZxJlfKuOfE/aZSECc9kPBWQUnN/RqcJeekbL0nADqnWuS3Sd8Pf1p0vg97cUUb39uOhIKQbEURt/g6mxELs+5y/YOhWCnT3JglL4OG6VExfdn21DhOrwBod9Pv2KEB8TjxY0M5SNuZWfdPflYJpltw0Y5vJMZShGWL2VExZlP0LBU0TuDSaq6v4jAmE+otgfrd4l3Jh6f+4fPyPGznzv4eDXRtm8Vy+4+inT+eA9WDkr2rT+jzK7hZZXZa4DEHxKIa5hIFurRcQoYAdHkoreecv5tQ7ltsXz8f8sYg5PCSPzkB5MGZKLfiOY/IkXYnX5I+sNFT/fH9KVFnabHTvHfVpOyoKBf4Q1BkTOJllxH89ycRJ76B6rE1QXUmYF7hq5SWJ6+VBiykq1fazp3iT3j5ZvsnOsN/oBron2lof4BESAnPLtR9bj/cUfJNdhDFcfr3d1uu1DoqtiX+UyW+l7dFCuPrw0Vr/mP7LJ15ZEKdMDzEqcmxFPe8Xz7LTzYHtk47x/BHtsoeihnL6pIiIVW5lFy5I/nfbK0iHFx4p7lHSGJ3UckCN0dZ7zMpzPEexl6tuqi4ntKbxNIDWch0P1m6TNmnQMQCaQL6mU/bCn8WDFKDx4mvtZ92B0yZtj3bY7VBURcJx1XucYQwoSH/GSg0kq97U58a8tMLdeOf5yFcFXF11yvz8pk1Pt5MiUdWhzMRoPYhRyZ+IsQhPz9nTig2/4bSOjTuCZTZIPGKCtkcFqWXDebJZD5Etrc5OpPmroJCzhOxBpj95moDaQ/R0T5RSvaLUfttE6gkqhL3b/xxvtVuiXyrbPDtVJuMpOnUw8uAshMiunMgr5lHl5tkWSr/I4HBk/uSEBgWgdEbor7zhqTDbim/a0smT5sqnEjkrmKfNO9bq+O6/MHGjMYQy7+oBb64nrUmsY0MK/ Tz0X5ekJ t2UXdQ4FLyg0B4X+665e4QSKS559XRItHOwavlkocmglMhcSNSmmJWmwoxWMPTLpPYR8xDfiehWcCWuMqRX4QrDO7mslxWbLMQMVu3UN87mLZ64xiOj1LHPrXZoHe/mcjRWSMSRi6Gki+z/BtaPy3dlEE1roTy7L6j0YA7ELJ6hMLEiJrJTdc1MrlhUrW153vX3Lce5Jo2bd4Vq8vNrZySan2BFGeOVMIQvz7Ij4nj1OuT54qiU+It+cadNvHIruQvloyTyjdVPEErBg23aDl3jJV5/9VPaKTSR4GhgNynVFdf6Pu3aTqdNKwNHySMDVBLXgCoEzAy0nS/DtpSZe6EqCJT+uPSy5pk9ZUB024wOpRaZjqJDhzd28p/xP76xk8ys25uxTrZ4+cpfP/O83kNjFl85Wu4XYN0TAVy/J/Fe1UVHhReCvzn0n6BCW+eQEe8VUR9Fb6Fe1M+VIos1xwcFun8EBHHV8paULSK+62ltKRzp9noVQ5MO6CelGvrymdg3MQZRb1FQ8JEzrIhcQjEAxh1fKaUFX4StjrR4khmdQQbdiTrFqV0Z19TmWbbxSSJI4CGh8emoAfZs3s94+jRqogEOKB71ug9nYgSjBCBvDhnvDjFAqI1Pai3f3R/Bt+VuIKLAe+wkP04TDmZG/yZ6mlJvA7FuTDNPC98X9GUivljdyBDmzwkKo0lr/bPMDh6joxCaWIW3dmBnaGWyvtDbIddU/98cpmQdTTitwkiF60Kv/XFedpVoJQxw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: --MP_/L6u2AePrH2ig2h2isRGtuja Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline On Sun, 2 Jun 2024 20:03:32 +0200 Erhard Furtner wrote: > On Sat, 1 Jun 2024 00:01:48 -0600 > Yu Zhao wrote: > > > The OOM kills on both kernel versions seem to be reasonable to me. > > > > Your system has 2GB memory and it uses zswap with zsmalloc (which is > > good since it can allocate from the highmem zone) and zstd/lzo (which > > doesn't matter much). Somehow -- I couldn't figure out why -- it > > splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone: > > > > [ 0.000000] Zone ranges: > > [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffffff] > > [ 0.000000] Normal empty > > [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffffff] > > > > The kernel can't allocate from the highmem zone -- only userspace and > > zsmalloc can. OOM kills were due to the low memory conditions in the > > DMA zone where the kernel itself failed to allocate from. > > > > Do you know a kernel version that doesn't have OOM kills while running > > the same workload? If so, could you send that .config to me? If not, > > could you try disabling CONFIG_HIGHMEM? (It might not help but I'm out > > of ideas at the moment.) Ok, the bisect I did actually revealed something meaningful: # git bisect good b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17 Author: Yosry Ahmed Date: Tue Jun 20 19:46:44 2023 +0000 mm: zswap: multiple zpools support Support using multiple zpools of the same type in zswap, for concurrency purposes. A fixed number of 32 zpools is suggested by this commit, which was determined empirically. It can be later changed or made into a config option if needed. On a setup with zswap and zsmalloc, comparing a single zpool to 32 zpools shows improvements in the zsmalloc lock contention, especially on the swap out path. The following shows the perf analysis of the swapout path when 10 workloads are simultaneously reclaiming and refaulting tmpfs pages. There are some improvements on the swap in path as well, but less significant. 1 zpool: |--28.99%--zswap_frontswap_store | | |--8.98%--zpool_map_handle | | | --8.98%--zs_zpool_map | | | --8.95%--zs_map_object | | | --8.38%--_raw_spin_lock | | | --7.39%--queued_spin_lock_slowpath | |--8.82%--zpool_malloc | | | --8.82%--zs_zpool_malloc | | | --8.80%--zs_malloc | | | |--7.21%--_raw_spin_lock | | | | | --6.81%--queued_spin_lock_slowpath 32 zpools: |--16.73%--zswap_frontswap_store | | |--1.81%--zpool_malloc | | | --1.81%--zs_zpool_malloc | | | --1.79%--zs_malloc | | | --0.73%--obj_malloc | |--1.06%--zswap_update_total_size | |--0.59%--zpool_map_handle | | | --0.59%--zs_zpool_map | | | --0.57%--zs_map_object | | | --0.51%--_raw_spin_lock Link: https://lkml.kernel.org/r/20230620194644.3142384-1-yosryahmed@google.com Signed-off-by: Yosry Ahmed Suggested-by: Yu Zhao Acked-by: Chris Li (Google) Reviewed-by: Nhat Pham Tested-by: Nhat Pham Cc: Dan Streetman Cc: Domenico Cerasuolo Cc: Johannes Weiner Cc: Konrad Rzeszutek Wilk Cc: Seth Jennings Cc: Vitaly Wool Signed-off-by: Andrew Morton mm/zswap.c | 81 +++++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 27 deletions(-) 'bad' bisects were where the "kswapd0: page allocation failure:" showed up when running the workload, 'good' bisects were the cases where I only got the kernels OOM reaper killing the workload. In the good cases the machine stayed usable via VNC, in the bad cases with the issue showing up the machine crashed and rebooted >80% of the time shortly after showing the issue in dmesg (via netconsole). I triple checked the good cases to be sure only the OOM reaper showed up and not the kswapd0: page allocation failure. Bisect.log attached. Regards, Erhard --MP_/L6u2AePrH2ig2h2isRGtuja Content-Type: text/x-log Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=bisect.log git bisect start # Status: warte auf guten und schlechten Commit # good: [88690811da69826fdb59d908a6e5e9d0c63b581a] Linux 6.1.92 git bisect good 88690811da69826fdb59d908a6e5e9d0c63b581a # Status: warte auf schlechten Commit, 1 guter Commit bekannt # bad: [a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6] Linux 6.9 git bisect bad a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6 # good: [830b3c68c1fb1e9176028d02ef86f3cf76aa2476] Linux 6.1 git bisect good 830b3c68c1fb1e9176028d02ef86f3cf76aa2476 # good: [1b580c9bb63414903cfb414571e92e4fd36d6969] Merge branch 'samples-bpf-remove-unmaintained-xdp-sample-utilities' git bisect good 1b580c9bb63414903cfb414571e92e4fd36d6969 # bad: [5ab500d6f9f50d8246865a2ead85d0e88ea30004] Merge branch 'sfc-implement-ndo_hwtstamp_-get-set' git bisect bad 5ab500d6f9f50d8246865a2ead85d0e88ea30004 # bad: [cc33a80b816406f900a53c7f98a50f6eacdd2e31] MAINTAINERS: Maintainer change for ptp_vmw driver git bisect bad cc33a80b816406f900a53c7f98a50f6eacdd2e31 # bad: [4debf77169ee459c46ec70e13dc503bc25efd7d2] Merge tag 'for-linus-iommufd' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd git bisect bad 4debf77169ee459c46ec70e13dc503bc25efd7d2 # bad: [1086eeac9c333b6db6c98594f02996c8261c60c5] Merge tag 'lsm-pr-20230829' of git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/lsm git bisect bad 1086eeac9c333b6db6c98594f02996c8261c60c5 # good: [0fc81f376468ce67d70ecd1016dfe901a51a39f5] Merge tag 'regmap-v6.6' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap git bisect good 0fc81f376468ce67d70ecd1016dfe901a51a39f5 # good: [bd6c11bc43c496cddfc6cf603b5d45365606dbd5] Merge tag 'net-next-6.6' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next git bisect good bd6c11bc43c496cddfc6cf603b5d45365606dbd5 # bad: [5553b15a4bbba8039e1f31b63642048286f540dc] m68k: implement the new page table range API git bisect bad 5553b15a4bbba8039e1f31b63642048286f540dc # good: [0c2e394ab23017303f676e6206a54c54bb0e3681] mm: move FAULT_FLAG_VMA_LOCK check down in handle_pte_fault() git bisect good 0c2e394ab23017303f676e6206a54c54bb0e3681 # bad: [3f32c49ed6f15c8412a8abc93a92c4b37e6c4592] mm: memtest: convert to memtest_report_meminfo() git bisect bad 3f32c49ed6f15c8412a8abc93a92c4b37e6c4592 # bad: [6ad243b83b5094026fdb3171711ddb25246b3d8a] mm/damon/sysfs: implement a command for updating only schemes tried total bytes git bisect bad 6ad243b83b5094026fdb3171711ddb25246b3d8a # bad: [68af05143fd4b49d4b12eab8d63c91ffbc7c4e5e] kernel/iomem.c: remove __weak ioremap_cache helper git bisect bad 68af05143fd4b49d4b12eab8d63c91ffbc7c4e5e # good: [27af67f35631ac4b61b5e4455b44c9aee8d2cc4b] powerpc/book3s64/mm: enable transparent pud hugepage git bisect good 27af67f35631ac4b61b5e4455b44c9aee8d2cc4b # bad: [34f4c198bfbe86612c368eb122002787acecaa93] zswap: make zswap_store() take a folio git bisect bad 34f4c198bfbe86612c368eb122002787acecaa93 # good: [601f006fddc66e369fdac7c572f981eafd159dac] powerpc/book3s64/radix: remove mmu_vmemmap_psize git bisect good 601f006fddc66e369fdac7c572f981eafd159dac # bad: [b8cf32dc6e8c75b712cbf638e0fd210101c22f17] mm: zswap: multiple zpools support git bisect bad b8cf32dc6e8c75b712cbf638e0fd210101c22f17 # good: [6be3601517d90b728095d70c14f3a04b9adcb166] powerpc/book3s64/radix: add debug message to give more details of vmemmap allocation git bisect good 6be3601517d90b728095d70c14f3a04b9adcb166 # first bad commit: [b8cf32dc6e8c75b712cbf638e0fd210101c22f17] mm: zswap: multiple zpools support --MP_/L6u2AePrH2ig2h2isRGtuja--