From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BDA5C25B78 for ; Tue, 4 Jun 2024 16:12:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01E406B0083; Tue, 4 Jun 2024 12:12:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F0FEF6B0089; Tue, 4 Jun 2024 12:12:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD77E6B008A; Tue, 4 Jun 2024 12:12:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BFB386B0083 for ; Tue, 4 Jun 2024 12:12:04 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4F96FC0EBB for ; Tue, 4 Jun 2024 16:12:04 +0000 (UTC) X-FDA: 82193697768.12.1A24B69 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) by imf30.hostedemail.com (Postfix) with ESMTP id 679C48001B for ; Tue, 4 Jun 2024 16:12:02 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DKEXVjum; spf=pass (imf30.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.46 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717517522; a=rsa-sha256; cv=none; b=hz68bWw60xdY2B2AQ6ZQNltu4z4BiADoO2YG+MnSKm7yRHQE7s9uIGc+46RggPdb03s3S6 TOn2hkEnNO1VTNR1iGliiuukZfDQWAVDXbAQPKJdNWvbpFSylktTcrvzorHCp+HMd+BIh6 P985PMvUKNuGrlSxTVn1qK3/8Hc15Uk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DKEXVjum; spf=pass (imf30.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.46 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717517522; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7Fq1N08ityDodrU5UqN/tu7ksF/MqwNuqjUfkSs8E40=; b=zRAUiuJDz86+XuCL6nOXhBIcIRNvFUDEf+Puih89ZtBjuQKygAewj+5BHYVtobvqK3Smv3 g3jDzvdqhnnLsN9oY2i1I0iFUWW3K1VpDX4eYph8WzupJFABG1tVwHhh1dn8NU6ASy0xjW /buM0S37v+xxYY5e8/Lh0CeHq8NLsEc= Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a68c8b90c85so384225066b.2 for ; Tue, 04 Jun 2024 09:12:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717517521; x=1718122321; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=7Fq1N08ityDodrU5UqN/tu7ksF/MqwNuqjUfkSs8E40=; b=DKEXVjumx/l6L9LRweutlFXMY5CAlfRz1D9t5DmTG9YaKogkQMWklaRdk/xBvAU5Ml d3jjLWO4ZoESrIisyjK6prr7fpeVyXI6/n1aYw+HpKCaC4CUWZRf9yLx3YPfxevH7RIo GujH5EVvPOlt4RoREBrjRaoHyAZ2BjR0aZ6Sq7b9Vs4rPYjJSo5Gy/8ukF4QreNPptz/ y+FXgKnDewAGl0as8SrrEgb7eH6sjwTDK99TzxXQVffEV7jfJWDKjAAD/koN3SM53g7s lyIngYdZiOAZxjVjwlkPRPskFKmBHea5b1gJuR1RMPOYSSdgKJvZO3Kz+qaSJabnZmjR nM2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717517521; x=1718122321; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7Fq1N08ityDodrU5UqN/tu7ksF/MqwNuqjUfkSs8E40=; b=rsYhHdJ2BuWNqpwaSDq5dhfLxZu7PqltvrJtRpD6TF9fFYpyje9c2KTHZeWInMtlMd RsbplFG1wpipyQxFF1wfvao3zjz265XZAPb3PFEHGOns7PF6yxUfpqBlfZkw8rmCS34k 5tabfCvxlvnnydz3TM//bo+ylS20vBmFfDIcd7NvWUh0DJkumggjL+KWg5d46CsoVsGv +tgWqmBiXNFiT34ApIR9YbMZCQtcDKqVa2Gf9PVFJejAdcjMClFSvlWLWLk2KsIAcrzf nqGL/lf/vzoaUk49LkrY72vRg/fKHhSaoZgZdxoCN1RUS6K5p0Mat0TSvp8AXxoMlnz0 2THw== X-Forwarded-Encrypted: i=1; AJvYcCXPBeiqHcO4RZ7np0d0x3lrCcGDdbH5Xi2R1eKbI0saKwGHulleTi1Rl44JNHeKAQTq8ldnPow+GKi79nGkIjmQz3o= X-Gm-Message-State: AOJu0Yx5GP5pmhkOmsA+pTyNz9Q/3XIX6b3+4nMUJQfpllS31zOPwmbw ukyz7qr8MIBhUGlOWUijhujCEx9tpOuWt9dcEAkAxFHv+PE+3cHQSKn3L6Xq6CpVPtA55utWe75 F5LFvU4QG9uD5uQ+vzJIETnNVXAT9EjIFou/1 X-Google-Smtp-Source: AGHT+IHjToe9HZIMn9NzVD0RPeflJl/bU+DgMt9GMRZwk/Ij7exz/RMTsjf5+KctCPjgAFtdRjBi3oCz1G19/QxZrYg= X-Received: by 2002:a17:906:35da:b0:a59:d2ac:3856 with SMTP id a640c23a62f3a-a699f66643bmr380966b.22.1717517520321; Tue, 04 Jun 2024 09:12:00 -0700 (PDT) MIME-Version: 1.0 References: <20240508202111.768b7a4d@yea> <20240515224524.1c8befbe@yea> <20240602200332.3e531ff1@yea> <20240604001304.5420284f@yea> <20240604134458.3ae4396a@yea> In-Reply-To: <20240604134458.3ae4396a@yea> From: Yosry Ahmed Date: Tue, 4 Jun 2024 09:11:21 -0700 Message-ID: Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc) To: Erhard Furtner Cc: Yu Zhao , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Johannes Weiner , Nhat Pham , Chengming Zhou , Sergey Senozhatsky , Minchan Kim Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 679C48001B X-Rspam-User: X-Rspamd-Server: rspam12 X-Stat-Signature: cq88qi3q9x7z8npkkqx6i8d9g91y4js1 X-HE-Tag: 1717517522-396927 X-HE-Meta: U2FsdGVkX18L3fc32HHMg7zx/QkyCSZiMTEr1mq3G+W0IapQT+R20/a4DnIy7EXVhjfnXsSu2TUITP6SRiq8JNMjWlNm21MtFYr/uzK7e7m2obvfAgRxNKhbOfzSRm5xp9/JMJ3X4eNWqzo1MCgrZt/YUsbqe+kjun6sPFuC0EuVYteOmRJfCKR/whjX4VR4zr7fbSK3EM+50tP+A+++BVGRA1e1BzlSuhWVq0br/WzynrVLF55yGDv2XLr6YQKCf/MpDdm+XkwoQ5iu1zWXQ6Uo4QMdloL09CYZHE39TyYrwaQBpwDUiCRbivBYgZ66Y5JkTs6uOpvlPBuUP/vjG4dJaMlejqS/dAwe9Y4uWmbInb8NEyuqOT/NncjSXXZr9QhOVJFKiDzUkORaiNBeT3Q6ulj+q3H4i4tGJG+dy3NCozZnKlIBuK1BQjeu+PKA95i53/UGk51sEQBdhOa0F6zsdpRia9rVc2LIUmGvkuAIrZMMbrk/M0QGqsyzxInlwIIMiGC3N34hm2ik5lI7Tzg97yXyDnRDPoxlVSItrrX3Wp/EOwVOud/O7wcVes1hxHbrbp/TaSl7yErhq48AHDseASyUwiZ/TPAiNsMe6GjedOlTDLpkpqS+oJw38gGnvS5IBafia+d8DBSgC1K8xXEPg5peOKc/LLU7HUtmGInqFfnNKwKhweqfGxaTV5Y9q1hzbZJhM7w1zNabiLHWq29HDZzMBw+4tQAiTPGDwh+yklvtY6bgM7z9SDpfIa3B2SfbzZPRPN6WRT3ZWtYVD7Wjic3Lygc9zJqj+dqOv8mmgQnPTKiIUB8X0ewa8RQac4zaS2Q1tsaS1WVDj4d9UO1JO6GeS1BJnIs+tVn1bad6d4FvJMX0LU6wCo3Pmu1wM3aS8L/Cd5IYfpD0U7HiBwhJOmLqkqVS3GDUsV6EbqoP58mcpTWUe4Ga5kyeRQkg82fehTNq/FRpGQZbVAV lbh/aycp 8MkaFaExb8y9Ni9imM2HZRRL+CGVP157A4EiTD40lp41XL0B+2HMyde8napywH7rIbfe/+/lSVlZgAIEGv3Dg5+qVeAADlIRheoMt9rZbwf99hrbhG/4+ilhyd8wJAuI/TuKZKXJHz0aNrKL5uesykd5H0uDFtIzJ33qgZv1r40uKQXj+nZzIoxvVvkNaI7JNVWOcFlKPCzNNdix6buDCUXSBH7fDUvX4XWP7bbAkbyJfps2LTNCJFyQ38jSYjtdVmI5zcJIjsTFK0ukc1hDHn+SOiECOrammZuWABbW411hxrUCWkgyC4mP4VaUqyAmS/FhvzNWsh0wvTTdb2gqxTqiF4y5sv7PH0JrPS7Ey3l+8EqA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jun 4, 2024 at 4:45=E2=80=AFAM Erhard Furtner wrote: > > On Mon, 3 Jun 2024 16:24:02 -0700 > Yosry Ahmed wrote: > > > Thanks for bisecting. Taking a look at the thread, it seems like you > > have a very limited area of memory to allocate kernel memory from. One > > possible reason why that commit can cause an issue is because we will > > have multiple instances of the zsmalloc slab caches 'zspage' and > > 'zs_handle', which may contribute to fragmentation in slab memory. > > > > Do you have /proc/slabinfo from a good and a bad run by any chance? > > > > Also, could you check if the attached patch helps? It makes sure that > > even when we use multiple zsmalloc zpools, we will use a single slab > > cache of each type. > > Thanks for looking into this! I got you 'cat /proc/slabinfo' from a good = HEAD, from a bad HEAD and from the bad HEAD + your patch applied. > > Good was 6be3601517d90b728095d70c14f3a04b9adcb166, bad was b8cf32dc6e8c75= b712cbf638e0fd210101c22f17 which I got both from my bisect.log. I got the s= labinfo shortly after boot and a 2nd time shortly before the OOM or the ksw= apd0: page allocation failure happens. I terminated the workload (stress-ng= --vm 2 --vm-bytes 1930M --verify -v) manually shortly before the 2 GiB RAM= exhausted and got the slabinfo then. > > The patch applied to git b8cf32dc6e8c75b712cbf638e0fd210101c22f17 unfortu= nately didn't make a difference, I got the kswapd0: page allocation failure= nevertheless. Thanks for trying this out. The patch reduces the amount of wasted memory due to the 'zs_handle' and 'zspage' caches by an order of magnitude, but it was a small number to begin with (~250K). I cannot think of other reasons why having multiple zsmalloc pools will end up using more memory in the 0.25GB zone that the kernel allocations can be made from. The number of zpools can be made configurable or determined at runtime by the size of the machine, but I don't want to do this without understanding the problem here first. Adding other zswap and zsmalloc folks in case they have any ideas. > > Regards, > Erhard