From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F824C27C52 for ; Tue, 4 Jun 2024 17:34:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8972D6B0093; Tue, 4 Jun 2024 13:34:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 846626B0095; Tue, 4 Jun 2024 13:34:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70DCF6B0096; Tue, 4 Jun 2024 13:34:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 531176B0093 for ; Tue, 4 Jun 2024 13:34:50 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EE4DCC1017 for ; Tue, 4 Jun 2024 17:34:49 +0000 (UTC) X-FDA: 82193906298.18.92E64E6 Received: from mail-lf1-f49.google.com (mail-lf1-f49.google.com [209.85.167.49]) by imf11.hostedemail.com (Postfix) with ESMTP id 0F7F240012 for ; Tue, 4 Jun 2024 17:34:47 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rvT3IMSK; spf=pass (imf11.hostedemail.com: domain of yosryahmed@google.com designates 209.85.167.49 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717522488; a=rsa-sha256; cv=none; b=7QDmb/qhqVJtg3LFTX7kSCcc9XA16GI4XWpC9HzmXJGQ79r4QNtJu5fTf2ZIE/687xnujt 6wAL/THRYRP27pi2nYfXcD4hUPGtUyfVDSVEE8VbhVSfJD3Ys+bcMkkDAy5+JZXqKFvwUy 1CBnlvHczAau3ZxBWUCUHu7Wt3YukGc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rvT3IMSK; spf=pass (imf11.hostedemail.com: domain of yosryahmed@google.com designates 209.85.167.49 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717522488; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FSAsMIUgpkuY+akx1I0POUHLWSTt61sSZO6HEcvaGx0=; b=xhelTWNjpTdTgrQlWbgiH0hJam1espD81kCveDZL4VvVCdlr36JJwSr8iPZsBuz6Sqw+iK 8htRxTVqujH11MbCjKEvNyGAXaEEjiyHuAQ//jZlnyO1BV5Ope/8c7f0/atGJeelgEm1kT yJ2rEvMtBBumzUfyolMdKZa8iZg2DZw= Received: by mail-lf1-f49.google.com with SMTP id 2adb3069b0e04-52b950aa47bso1864173e87.1 for ; Tue, 04 Jun 2024 10:34:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717522486; x=1718127286; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=FSAsMIUgpkuY+akx1I0POUHLWSTt61sSZO6HEcvaGx0=; b=rvT3IMSKAAyshGVtVdir69i78QBH/7Gi4G/LBYYCpjxjcXVgirFlVGQNGtLQXbGjW+ orcEjoaSkgMU8WOs9CGyEcwCUF++FSkLSzophzZ06Vr1Zg50/4y6/OHpBkn9xUH8nPW4 CvXv23vhMMS2qzrHEdGbXx9sGdwZPoyJoGNsdxDwnOUorTvfL5doC3aJ+lTe069eXzSK mj1Qw5kF+UkZFpgSbzZMtFxZLo4o4Yz2LchU0ZIROdcw3koj0YbdWUYQjCYFVpmeRXjx AXq8QIyyl7FNYEI4o3Hm7Mc8AODK3uOHf2m3h4FDx/PY0XZ5N5GIEdHqwdg+LIY5kuZb PBLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717522486; x=1718127286; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FSAsMIUgpkuY+akx1I0POUHLWSTt61sSZO6HEcvaGx0=; b=JbFnqTPi99Us90Attwa0oFxJK0Kb8Tz4LH2nK2OEn7CL99a+Xf42WPYqZEo5u5GjGQ v0bTSS4TpLEPRD8QNdvw9Jybh2c7r6yEZARYqQLUgH1inXfJANKOi44DdGSY031bfHoL qF3lrea8gjtI2/9WYEBk9DthPFsXZ9zQC/PblK5LhuupBHpKPl3hhp1SWukiYl9K/edN jpk91OyLlCs6Pj1M/SNgEy7otbuTDqv3J7+2XzCfCsgseaDD63Ae76pLkmut0CIsAon6 cwoODMZssomdmZi53tj2uTPUTm7+pHn2eg6pVcS16P7iyemVOJA0wbenfKGErhipwJCW VxPA== X-Forwarded-Encrypted: i=1; AJvYcCVpqcxNJ1hudXcOfplQ4ptEipNIaffQ2OryITeDFFr6y3HxSxZCnXYQx695enfV/Ms91O07fvXpDuaPIGC38BGbkEY= X-Gm-Message-State: AOJu0Yzr4pi3p+xYCPpzIy70zHJTMDUQfAt89cT6L5EPUWMdiTF1Ea+X 3kXXDSSy23gfNXe4cz2j9ZcUfTNe3r1aGqzqfMgWO88wmG1bQw2/cZnvL0S+xX8VKTGXBq9Tpv1 orDDUXKGslFOPjJO4b61VLG2pIDlJzlZNg+A0hl8/HD5btBZN+AgO X-Google-Smtp-Source: AGHT+IHW7hvfcVnmpj1/QggPMRaKYlE7gegvyFwo9KyDFTK+obmZomYA3kKP97Zw+Ic1XOVGi3IkSE+oKfX0RqFKMmE= X-Received: by 2002:a05:6512:3da3:b0:51f:5872:dd8c with SMTP id 2adb3069b0e04-52bab4e8a85mr148583e87.39.1717522486012; Tue, 04 Jun 2024 10:34:46 -0700 (PDT) MIME-Version: 1.0 References: <20240508202111.768b7a4d@yea> <20240515224524.1c8befbe@yea> <20240602200332.3e531ff1@yea> <20240604001304.5420284f@yea> <20240604134458.3ae4396a@yea> In-Reply-To: From: Yosry Ahmed Date: Tue, 4 Jun 2024 10:34:07 -0700 Message-ID: Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc) To: Yu Zhao Cc: Erhard Furtner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Johannes Weiner , Nhat Pham , Chengming Zhou , Sergey Senozhatsky , Minchan Kim Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 0F7F240012 X-Stat-Signature: byfafxbwj1ygqdef3j73qmfqexaa79m5 X-HE-Tag: 1717522487-984878 X-HE-Meta: U2FsdGVkX19nEGkvCjon8LnLmlWhFQ5rtwt5EWMFC1FEQtcsZUXjFZQQxdl7duKefpbWyhgAktfNRcYIVlp4qoc83enen9I0o7us98Kt9Cv17mnzf/3iJdI/qHzjwuMfJy9XIAe6tu+Qr/QXp4paBOG2URzrdtDTYgSthHo9/Lfc9lngnkGJHG1UWHBGR3Bio6uanPHejekCPk7M+gtCq6ICM/yPpKs2/y7nF4x60ehpxzHKPtHWiNnsOICbiaQ3sRpAr7sZuxS+/t7nLCScEfSkfP4A6fnkojgJFJb9iZRycttkkZlJ+D/MPs18rSKHxCrE9gUrz2scYHcTeG1TDrKei+BZi7Ph0VLWPDIakoqPA36oHSzdy6Hd3gV8E0xbecC5d/gsWKGL9xSBNfrWi/EiaMlIrZcEJXtGlu1T82jAhz8LBPOk1fijPFyg/OnVtkql6ewzwOPQ7vOLwnhh7tOonpYlPS0R0khnTfr64012njDdHN7fxhR9nRMj6Titz9DZ5ygPiB4Hf7z78nUPq2I/SC4kOeUt8nkz8t9XLUkLsDx7UloasrT1cbHaiBYk3rMe+VcDcOaPCFe2ejrl7OohXVYbpkvCLsip7JyY8h8YCfXfvG9AB9IfCufHFPYkD8yHH2s11tMmtNpJBLHLfjmPwBSuCbN7i0sUsmdIezfVb0GWc3hZfArhgY04juTakfLudnWYRZr03xtJUeaLHtx94zpYCeBoS4l8o/daG0ASIElSYhugvkHwI/XqQrbBUKWR4UncAC6fwYK2HYhUkkl5IvoIYVGKEeGHupBN5X6/ILAmekfIcqvUKu3T1thD0L5jJejaz+btqOZiGjO2p9TZxYoU6vEeitCtUXYmT11fGTvmPCHybRqJteeZjwVbVU3xM5tgLPpbI8ioGT1jd19gVVItl7fRIqNvVyREUh9iZAs3rVQHDnikja/FLDQjEB/lCTL5rDj+iUMsZVY 3JB07wVr yk6Lb+7UZVbPLGDMN1Hpy9YIKtnU3VOY1a4QEnbB9/zSzrPKcnL3mmxDL/WLTfBKBV9pjrPuZnzxKg0cE9txjOizpyEBhMav2mg005Bp5p7cKTvA0Qm0cCFDjEtUhGXQ4beU4G4GkQQ6TatmuMKKiWUhSE2/qEZ0ZcNHvfmuO8K9R4j8ZCZa2Y0VMgnfNCdKEd5/Lm7MHlBWk94AJXvjYk8/ujIP/s2M1nj8fqkzj1B9prtg8gqILDKrfyHqL8W5ayoBsBvHJRMqXFu47FyGr7DhSu0D81npvzOEq1fmlYltQ9cj0HDwVo/vyt9dtgNrrmRnyKIjz5FOit8Ujx56e36TBRIsEVkEDMBtW86XBN7QJtn4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jun 4, 2024 at 10:19=E2=80=AFAM Yu Zhao wrote: > > On Tue, Jun 4, 2024 at 10:12=E2=80=AFAM Yosry Ahmed wrote: > > > > On Tue, Jun 4, 2024 at 4:45=E2=80=AFAM Erhard Furtner wrote: > > > > > > On Mon, 3 Jun 2024 16:24:02 -0700 > > > Yosry Ahmed wrote: > > > > > > > Thanks for bisecting. Taking a look at the thread, it seems like yo= u > > > > have a very limited area of memory to allocate kernel memory from. = One > > > > possible reason why that commit can cause an issue is because we wi= ll > > > > have multiple instances of the zsmalloc slab caches 'zspage' and > > > > 'zs_handle', which may contribute to fragmentation in slab memory. > > > > > > > > Do you have /proc/slabinfo from a good and a bad run by any chance? > > > > > > > > Also, could you check if the attached patch helps? It makes sure th= at > > > > even when we use multiple zsmalloc zpools, we will use a single sla= b > > > > cache of each type. > > > > > > Thanks for looking into this! I got you 'cat /proc/slabinfo' from a g= ood HEAD, from a bad HEAD and from the bad HEAD + your patch applied. > > > > > > Good was 6be3601517d90b728095d70c14f3a04b9adcb166, bad was b8cf32dc6e= 8c75b712cbf638e0fd210101c22f17 which I got both from my bisect.log. I got t= he slabinfo shortly after boot and a 2nd time shortly before the OOM or the= kswapd0: page allocation failure happens. I terminated the workload (stres= s-ng --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB= RAM exhausted and got the slabinfo then. > > > > > > The patch applied to git b8cf32dc6e8c75b712cbf638e0fd210101c22f17 unf= ortunately didn't make a difference, I got the kswapd0: page allocation fai= lure nevertheless. > > > > Thanks for trying this out. The patch reduces the amount of wasted > > memory due to the 'zs_handle' and 'zspage' caches by an order of > > magnitude, but it was a small number to begin with (~250K). > > > > I cannot think of other reasons why having multiple zsmalloc pools > > will end up using more memory in the 0.25GB zone that the kernel > > allocations can be made from. > > > > The number of zpools can be made configurable or determined at runtime > > by the size of the machine, but I don't want to do this without > > understanding the problem here first. Adding other zswap and zsmalloc > > folks in case they have any ideas. > > Hi Erhard, > > If it's not too much trouble, could you "grep nr_zspages /proc/vmstat" > on kernels before and after the bad commit? It'd be great if you could > run the grep command right before the OOM kills. > > The overall internal fragmentation of multiple zsmalloc pools might be > higher than a single one. I suspect this might be the cause. I thought about the internal fragmentation of pools, but zsmalloc should have access to highmem, and if I understand correctly the problem here is that we are running out of space in the DMA zone when making kernel allocations. Do you suspect zsmalloc is allocating memory from the DMA zone initially, even though it has access to highmem? > > Thank you.