From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F458C27C52 for ; Tue, 4 Jun 2024 20:56:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 032106B0088; Tue, 4 Jun 2024 16:56:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F25EE6B008A; Tue, 4 Jun 2024 16:56:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC5196B008C; Tue, 4 Jun 2024 16:56:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BB8466B0088 for ; Tue, 4 Jun 2024 16:56:30 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4433DA0BC3 for ; Tue, 4 Jun 2024 20:56:30 +0000 (UTC) X-FDA: 82194414540.09.4B07ED0 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by imf28.hostedemail.com (Postfix) with ESMTP id 79A72C0014 for ; Tue, 4 Jun 2024 20:56:28 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MINaBCxW; spf=pass (imf28.hostedemail.com: domain of yosryahmed@google.com designates 209.85.221.44 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717534588; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D/2NrBR2dMh4jxaJHvIIkbx6ft/lfiJX+xOAj1QR9Ns=; b=40mTD5qR87jcxoJPeK8Ks1zgP+t9QLJeHDNzDPsnwJPeJB7mDH+PIcyQH0tJu+kZqBrcaV PHa7ajz/Ik0WoQTSEEMWWfbsrQaDM5PdlH5UbO2k98vlVPgjyHlTwROpi6z6L+CGJobKrw ZOcL804AXMPoRWExvfILKC7eDH03BeM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717534588; a=rsa-sha256; cv=none; b=wSFYYDVCTagPMLHrg3T/Ts/IegGxf+vQhTvLBouxAKBFOSpoP8JGOiOFU/uA4PV54u7cOT 3SBji0TWwBLdx7XkI+2R7JnoK9dcnlB9NqfvwMVf39Ev5vgdySEBMF1Trr8apu+Y7cgC9L elEoTeQGqTSiOkhiTcLm/eeQdP/SEsQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MINaBCxW; spf=pass (imf28.hostedemail.com: domain of yosryahmed@google.com designates 209.85.221.44 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-354b722fe81so3953307f8f.3 for ; Tue, 04 Jun 2024 13:56:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717534587; x=1718139387; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=D/2NrBR2dMh4jxaJHvIIkbx6ft/lfiJX+xOAj1QR9Ns=; b=MINaBCxWtANqXFv3nTNTMYj54Y1JkY0IWPOOtpTfJPAPc8uYf/34zV804lNKyLHgwZ 2vzCtuWxms0VUmx4Oq6oL2+i2qEwzTCBp2uUw1EMv86S9wrrQ97vavgTDdToMP6PsV/a OueiMV5sFIIOdfC+Zk7/xkflmE+lwu5zgmMUbb6wXNQ8ItRS4pSvy4G4sUvVfQ8vAZSK xbI+4YkXe+1Q70KPC6osh3upjDfo0dJyhQJmrI6oRto0AXNdhXQqFw0HpWFDU+vsI2Cx sMlHio29NpZCUGjzhVhp0iBrbuhKe7ipgKofjfk5O7WvCQWNPJOMj2ZAVN8Azf3o75mt 1P/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717534587; x=1718139387; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D/2NrBR2dMh4jxaJHvIIkbx6ft/lfiJX+xOAj1QR9Ns=; b=ScIBwTqAH3yZX+KWRdrsGhDirqPxlxb1OH0p09BCeJ/E3jTh9KVZVF0ZxNq7Q42IOm pkkGHt4GBbw7E79dP9aNNgCO452Lsh2TKxrpfRnmOypsCFXbx0fBp+6yJBhnyx7BNpje zYbp63X9YraYQykAjUB29/4/GvYA3d32T6wXENN/urrLkyPrndJgUL8+PhSQKH4O4jm4 Jx5mMs6mcVXeN8+j1dDXeEFNODdEGItttti3RvNKaWcn2YjhumgE6jn9Piav2AHo1W93 xl+deNgQj0POulmJO8gK7oRxS+IZYLhW4ZvDHOGlRkghbVRU0EFodtNCNmquMtQ8d22a hDlA== X-Forwarded-Encrypted: i=1; AJvYcCX3Xp9B08JdNuUsJwB9d6rXvuLiPtx784GVw1jVl0i3qjDaaj9ZRIFTJDKtQXEOmH1eoMDxGzLnKowIUbfjjRyV/TI= X-Gm-Message-State: AOJu0YyBFHxv98Lc/hlPAdTZuCte4EI5uW8HNTyQNuF6J+uuRLulTA75 qcxwpYbf0D6h226r36886qLM3XLMjV8+jCpJrM5cKMipEd7FBgzer2nHiClcHAfrcWU01UZN6Eu tt0u3RglO69fgIXvwi6MQm5z6g89X6/jt8HaM X-Google-Smtp-Source: AGHT+IE6JmQAll+GNDs0FWYMgmBviEFJQQ7PiqrGzd9Dp0vvE7AUoPNGThQbFCpeMIguYSHNdWujsqidz+e0T1ALypc= X-Received: by 2002:adf:f6c8:0:b0:355:41b:d391 with SMTP id ffacd0b85a97d-35e8ef93cc6mr429509f8f.65.1717534586541; Tue, 04 Jun 2024 13:56:26 -0700 (PDT) MIME-Version: 1.0 References: <20240508202111.768b7a4d@yea> <20240515224524.1c8befbe@yea> <20240602200332.3e531ff1@yea> <20240604001304.5420284f@yea> <5c69cee6-f101-4c86-b38a-7c5c8679ea9b@kernel.org> In-Reply-To: <5c69cee6-f101-4c86-b38a-7c5c8679ea9b@kernel.org> From: Yosry Ahmed Date: Tue, 4 Jun 2024 13:55:47 -0700 Message-ID: Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc) To: "Vlastimil Babka (SUSE)" Cc: Erhard Furtner , Yu Zhao , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: bs9xgnq5sgq363ni9xmroykdcmjmhrg4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 79A72C0014 X-HE-Tag: 1717534588-613616 X-HE-Meta: U2FsdGVkX1+Dc89dU5IFebVRYKny67h7VW628h7kIRMFExTEShVlUPr1Lnw8VmjDhxQX3i5bHuSi0Tl9EsyqxN2PsIWQ4o9ZBh8DWzMAdg67CZs9BpAB+8w3P7Cd6hH1VnvK1JXrlVcbchc+VxyFg0PK2al4fU/7EsC6F03ye4RuMkhRkD0Ht+DPyijDhwguqHBzzf/TYBodP7hevS4Z5RqQGRmjITespN/YBPmXWsnypT8wakaLMtotUXnDpSDl6PILjHuvMWiNMgAvmg935md4+V9FwIyvgjlvpUXbXAbJZA7lqcGbMCnk5GD6KN4SYtctY2t19XIiGZ6GsevQrgEBY7fdFccTsvq+BcNdBKbe2TsWpGfxTpJEO+DOoOQGp9cK8ENRxaChlGnLgW8GXkmDa0KBndeOWmBLil/IKKM9hvFDPw46WeZqHf5p3PBpAMIDqhCsAadqSWlPsSrX9m2TZSmiovXLSF9U6l5kp2LhWxyJCitAcC65sVOcQv1nAVbZ6QZvf92vRESS8AVJS62i71OQXu8HjTVoxsU2izKN669+QcHTkRwt/lRMO1rnQTdo4jlv8+sxEFSgZqebffCjl5S4w9GO7N2JoygSxAGBLGo5rZvChZGXNGPESA8usmuMpvR+EJBCkvBhDlKihH5RQv/9EScEw9IDVVlUS9Sw0h4FYUoCDazyatsxHs30Q4oR2sSCQvoog9nCVEHegLuGwjBU13xzo41579YGZnNeNHorZ+jAugYc6mqS2ytYUTWSPcOgG3FEYGAQJ6Yj6lwS/FAy7RylCOxSbyVC+M7rYnLNhyI3iiZWBDkZDYIgv36sON9Uf1Kckw3LVGgIzdHnpW7cAL6kJEjVo7TN8IXYogN7lQrxV84oa0G90eOlP+hqPjsBxGdVP1Gu/BqK4E2rXGuPL4NKxiZM2xeuN/Z8ugZy7+WcbnKFSwgHEdq2bMTCR7CFtkS89BoHXkv NuDKxk+R 3TKIRX/KBZ8eEQkmGeymjynfjrv+DAE88iw9aEs4pKnjbku+Af7rVhjIjje5QIdM2q4i+tZXvIzy9l65FXXj77jFQbPDGMYf2wFW7rN0Gk/KtvGCx7qwy1C5DVy2bEBZXIsVTeTnuaNQV+4Oj1q8IZEk81+Ob49t/lCZegGgPhCQVZ4270WZgcoNM2FvuQBOdZFePc8riMeay3XZIo8t3uebt1+nIzsqWlG5uMbdVcRGuc1yU3KYpQ2vv0bjiSi9FeDrH2YAW9Qr6KvA8iH31tR3t+8LjbbDz7KjiJwUQzU5vrDjPmexTby5fkCjLre9+CXT7xDcaHPGtSbycFpDWHQv1lm1Esk5eKehf44wukECikffPqc5S0Av/oudV1BSh6OgSklG2EJavzkjjgKgn1+iTUc7hah49R9Qe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jun 4, 2024 at 1:52=E2=80=AFPM Vlastimil Babka (SUSE) wrote: > > On 6/4/24 1:24 AM, Yosry Ahmed wrote: > > On Mon, Jun 3, 2024 at 3:13=E2=80=AFPM Erhard Furtner wrote: > >> > >> On Sun, 2 Jun 2024 20:03:32 +0200 > >> Erhard Furtner wrote: > >> > >> > On Sat, 1 Jun 2024 00:01:48 -0600 > >> > Yu Zhao wrote: > >> > > >> > > The OOM kills on both kernel versions seem to be reasonable to me. > >> > > > >> > > Your system has 2GB memory and it uses zswap with zsmalloc (which = is > >> > > good since it can allocate from the highmem zone) and zstd/lzo (wh= ich > >> > > doesn't matter much). Somehow -- I couldn't figure out why -- it > >> > > splits the 2GB into a 0.25GB DMA zone and a 1.75GB highmem zone: > >> > > > >> > > [ 0.000000] Zone ranges: > >> > > [ 0.000000] DMA [mem 0x0000000000000000-0x000000002fffff= ff] > >> > > [ 0.000000] Normal empty > >> > > [ 0.000000] HighMem [mem 0x0000000030000000-0x000000007fffff= ff] > >> > > > >> > > The kernel can't allocate from the highmem zone -- only userspace = and > >> > > zsmalloc can. OOM kills were due to the low memory conditions in t= he > >> > > DMA zone where the kernel itself failed to allocate from. > >> > > > >> > > Do you know a kernel version that doesn't have OOM kills while run= ning > >> > > the same workload? If so, could you send that .config to me? If no= t, > >> > > could you try disabling CONFIG_HIGHMEM? (It might not help but I'm= out > >> > > of ideas at the moment.) > >> > >> Ok, the bisect I did actually revealed something meaningful: > >> > >> # git bisect good > >> b8cf32dc6e8c75b712cbf638e0fd210101c22f17 is the first bad commit > >> commit b8cf32dc6e8c75b712cbf638e0fd210101c22f17 > >> Author: Yosry Ahmed > >> Date: Tue Jun 20 19:46:44 2023 +0000 > >> > >> mm: zswap: multiple zpools support > > > > Thanks for bisecting. Taking a look at the thread, it seems like you > > have a very limited area of memory to allocate kernel memory from. One > > possible reason why that commit can cause an issue is because we will > > have multiple instances of the zsmalloc slab caches 'zspage' and > > 'zs_handle', which may contribute to fragmentation in slab memory. > > > > Do you have /proc/slabinfo from a good and a bad run by any chance? > > > > Also, could you check if the attached patch helps? It makes sure that > > even when we use multiple zsmalloc zpools, we will use a single slab > > cache of each type. > > As for reducing slab fragmentation/footprint, I would also recommend thes= e > changes to .config: > > CONFIG_SLAB_MERGE_DEFAULT=3Dy - this will unify the separate zpool caches= as > well (but the patch still makes sense), but also many others > CONFIG_RANDOM_KMALLOC_CACHES=3Dn - no 16 separate copies of kmalloc cache= s Yeah, I did send that patch separately, but I think the problem here is probably fragmentation in the zsmalloc pools themselves, not the slab caches used by them. > > although the slabinfo output doesn't seem to show > CONFIG_RANDOM_KMALLOC_CACHES in action, weirdly. It was enabled in the > config attached to the first mail. > > Both these changes mean giving up some mitigation against potentai > lvulnerabilities. But it's not perfect anyway and the memory seems really > tight here. I think we may be able to fix the problem here if we address the zsmalloc fragmentation. In regards to slab caches, the patch proposed above should avoid the replication without enabling slab cache merging in general. Thanks for chiming in!