From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52D2CC25B78 for ; Tue, 4 Jun 2024 22:18:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D1C2D6B0093; Tue, 4 Jun 2024 18:18:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC9666B0095; Tue, 4 Jun 2024 18:18:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B910D6B0096; Tue, 4 Jun 2024 18:18:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9CB636B0093 for ; Tue, 4 Jun 2024 18:18:00 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4C18A160C48 for ; Tue, 4 Jun 2024 22:18:00 +0000 (UTC) X-FDA: 82194619920.03.A4221A3 Received: from mout-p-201.mailbox.org (mout-p-201.mailbox.org [80.241.56.171]) by imf08.hostedemail.com (Postfix) with ESMTP id 4BD9516001D for ; Tue, 4 Jun 2024 22:17:57 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=mailbox.org header.s=mail20150812 header.b=Mz4zL+fk; spf=pass (imf08.hostedemail.com: domain of erhard_f@mailbox.org designates 80.241.56.171 as permitted sender) smtp.mailfrom=erhard_f@mailbox.org; dmarc=pass (policy=reject) header.from=mailbox.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717539477; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=guuxTy+CZVyMI/LzvDTj+NobCvDyE9ykfdNc4s7xxKU=; b=JTaVtOhLxwhGEGNONP1N9LG4FEgMHUijJbCDgokDgMnhLTgj3jQvT3TD+cfZ+JE0tZz2P6 roCpc+co21BhQK9h4tFXxGBfqpCYQwYyoMwdDnCO8R+0wVz16fo/1ou8lb554FSGk1x2Tz IAvU7LqIAnhsy83ZhvO+Y9PHSOVSTB4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717539477; a=rsa-sha256; cv=none; b=NCGial3U5V5R+MpIUgGG7knSLphhJRjfPcsM2/IC05KGUWqrVPiCtTPxlO0QfT6JuPWsAM YmYpaJSXbPDxEfZQdOva8ImK+qSA16bFRYlM93p9wHcw/mZ/RLylERSgJz78teymj45gS1 qznNmtmOdw7bZPA6Cm4nq4DWjOzmsiA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=mailbox.org header.s=mail20150812 header.b=Mz4zL+fk; spf=pass (imf08.hostedemail.com: domain of erhard_f@mailbox.org designates 80.241.56.171 as permitted sender) smtp.mailfrom=erhard_f@mailbox.org; dmarc=pass (policy=reject) header.from=mailbox.org Received: from smtp2.mailbox.org (smtp2.mailbox.org [IPv6:2001:67c:2050:b231:465::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-201.mailbox.org (Postfix) with ESMTPS id 4Vv4jD2Rznz9sSN; Wed, 5 Jun 2024 00:17:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mailbox.org; s=mail20150812; t=1717539472; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=guuxTy+CZVyMI/LzvDTj+NobCvDyE9ykfdNc4s7xxKU=; b=Mz4zL+fkVn0xahQamRTxt5EodXqPnJdgpFxGEsfrP3tDTxMI+LHndT9Z8o1KAcwBPp1B+l VU4Ka0eWCs5sSk2DCHoMRZ9YG/UMSnfp+i/eI0pTcb4jNdwKT8hV9RltP54v1PsltDNWnW 6yOJkScrh8ZisDXePiSziqOVyxJXk9+kY2vGYc6mtMtDzXUlLFkRvrxB8Xxcj4QmBcmW4n O8e/JjOBVk/A3feAyjNQev4hhGBaBkOZpIQihowArQFMCW77Un3k+b82g91js6V+DOQPW2 zV4WyuOhiKTPb711dsqV9bAIuljb1AvFtHgRM85SqodWENpIC8QQPhSwrj1JlQ== Date: Wed, 5 Jun 2024 00:17:47 +0200 From: Erhard Furtner To: Yu Zhao Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Johannes Weiner , Nhat Pham , Chengming Zhou , Sergey Senozhatsky , Minchan Kim , Yosry Ahmed Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc) Message-ID: <20240605001747.2d3e13fa@yea> In-Reply-To: References: <20240508202111.768b7a4d@yea> <20240515224524.1c8befbe@yea> <20240602200332.3e531ff1@yea> <20240604001304.5420284f@yea> <20240604134458.3ae4396a@yea> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-MBO-RS-META: ik41k7m3xcnx6y18u6tq8497mnd3iouf X-MBO-RS-ID: e7369f9761aeea9ca46 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4BD9516001D X-Rspam-User: X-Stat-Signature: 6uogsh5r3xsq1td4mgu6iswn174rkfo1 X-HE-Tag: 1717539477-375045 X-HE-Meta: U2FsdGVkX1/LvncOiqNl8qq5fVEnaQPv976595EPSTaZNUyaTwLVRoEwdKOjpgxUL23dX5ApvTt3q0YmhwjED2r/N+7fxWRfkSFmSvqjLBZdtdx4qnq3Kd+hmx0VSNlVcdYJE4KJ8x6evivTL0EQcJyOVnvl+zSDKkbA2maPaZZVm4mUTpWa6XnorCoeTHXirqB61MRNJqPMAMPw5ZFDyITRs0/KW6kayj2OxdaMqs/fl3+AorFXKK+CcjLMBVFzX0Uh/kg02LMQNcRDUtFccVg2Q+uv4E3X755uqf6ciXhIU8CYT30NlOVYZwlOrK9cSFxJjuyApJaoT1/TIMp6p/tjRFZHqy6yK4nKAhOzrY1jXvGCdEZyQPTzy/GzeBHs+N78fbc8HY36rurUoJsnLWeeGGM7m55lD66TnYDlSwVBf8JpgfUH0jbsgUx70FEwPRlrbDg+Kxn7yIt1O3Nc7ypf5ciakpX/9Cv6MODVx4hoylykCEPaUCI29FVE196KQyh5pApCuM4dBRB8mn/FW0/qonRNSwmE/3M85CJWpXwfvvRehK8ThwXGQqu5dtIoU7CVq38gkiT4aL7sFnj56QpemASgLik+cioYeQzt8qyRH2YYPqZqks7MnL6/zXfqnuXSKeB/AvBsDAlKOZHcdTH0Jhup+jZ0RCz6t3/6iXUODgAV18TNInpWVHhvs7cpZq9+17Msot20CfC29XhS1QT7AsBBQ0e+vmSZecIp+/3Oh/QFQVEFj/31qZoH7wqDVJeVy558QJATuB4BuScRkJ9cHo47oJosgywTDKdAbdCj/ZzeuZWtNSYQ8NIDWekXldfaK/P4QV+5/ooe7DR+7qEj9cbMG1CDgZKbBoD5bu8BCEYtgcjOWq1MVDAf8vwlSTHepYh6idVwHaNvVwAj40SLsPaweL9mioXlRRYnmbs5P4zrKYncJW7T1ao+XcoXtD1BV3kyr9xQkahCdLQ dSejCYgH AE8u0PPaWpXscedKucELg7AjHG54qHgxtuQJwYdTr2UMMEPt5HSKI03Xbjc2sldh+BES7Vf3fEoN4v6M5iQxHyHKu+IEpDOxgs1HS2BMy00epLW3nZ5oQDl2JqHIVeo7mBXIvenk1qaI/2sOzkdnbtogShTyVyCWv/jxfyr5/LUNRBAm5qL62mpqqOPCsyChERPVNhYv4SZ6N4rBojLloYq3B34zGSoJTF7C2EI1s0gHsV1GB0PQW8NYUAMio++czZyDdO4MdnMdlyOs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, 4 Jun 2024 11:18:25 -0600 Yu Zhao wrote: > Hi Erhard, > > If it's not too much trouble, could you "grep nr_zspages /proc/vmstat" > on kernels before and after the bad commit? It'd be great if you could > run the grep command right before the OOM kills. > > The overall internal fragmentation of multiple zsmalloc pools might be > higher than a single one. I suspect this might be the cause. > > Thank you. I used watch -n1 'grep nr_zspages /proc/vmstat' to get the readings and repeated this 3 times to check whether the reported values differ much. Bad commit was b8cf32dc6e8c75b712cbf638e0fd210101c22f17 "mm: zswap: multiple zpools support", next commit in git log after the bad one would be 42c06a0e8ebe95b81e5fb41c6556ff22d9255b0c "mm: kill frontswap". With this kernel I got 2440, 2337, 3245 as nr_zspages and the kswapd0: page allocation error. Commit in git log before the bad one would be bfaa4a0ce1bbc1b2b67de7e4c2e1679495f7b905 "scsi: gvp11: Remove unused gvp11_setup() function". With this kernel I got 25537, 11321, 16087 as nr_zspages and the OOM reaper. Tomorrow I could also check the .config changes Vlastimil suggested (CONFIG_SLAB_MERGE_DEFAULT=y, no CONFIG_RANDOM_KMALLOC_CACHES) and report back if that's of interest. Regards, Erhard