From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4909C27C53 for ; Thu, 6 Jun 2024 02:49:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 326076B009E; Wed, 5 Jun 2024 22:49:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 284ED6B00A0; Wed, 5 Jun 2024 22:49:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 126226B00A1; Wed, 5 Jun 2024 22:49:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E58ED6B009E for ; Wed, 5 Jun 2024 22:49:48 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 65D251C10EC for ; Thu, 6 Jun 2024 02:49:48 +0000 (UTC) X-FDA: 82198933656.01.F2003A9 Received: from out-183.mta1.migadu.com (out-183.mta1.migadu.com [95.215.58.183]) by imf19.hostedemail.com (Postfix) with ESMTP id C64CA1A0005 for ; Thu, 6 Jun 2024 02:49:45 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=b15hR+TW; spf=pass (imf19.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.183 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717642186; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5TRIXdZH0MFsDrtAUMxWXmV78YJWERsVBogU5sB6ock=; b=wDUynCMSX/drkXYT30ZkLRn0NdehmDu+3s2+5Ywt22g6QR+3YB9KHqjQUfXGyLcpf8miS+ YZs1kxK5wdCOMp5gda4aPudV7mAbPM1FbwrXyEkvGcxjAwIUW16iWQtmbxiHqvmo3uAkyP KodAzd4y9kjaz9rMGiZAlun8lbrIXm0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717642186; a=rsa-sha256; cv=none; b=sINY+BzcH5Ir1MkvX+ZhmvMgzj9eno0bTgGFbsjJ+u1Py0eEK5RDiYVJvFswmAqEES+DLT JmlHE24bjsNpOPr6KvX5XqSqHwJk6NDaUd4/zGdyGfnhtVfzXmcDG9Ppv7Q7XFvk6cBajw znBCvAnUMs+BYl9orgyw4ErsxSxQBuY= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=b15hR+TW; spf=pass (imf19.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.183 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Envelope-To: yosryahmed@google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717642183; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5TRIXdZH0MFsDrtAUMxWXmV78YJWERsVBogU5sB6ock=; b=b15hR+TWR7d88Ppv7+DQY4gI99mEhGw5y7HNfof+KNldn8mCJBaHtegIi55tjR8gLi7Fvh LzjNotdZuArhnxmgbjP0qzxxvOOs2urIoOC9W/bzm/+ieKLjcNz5xptVOgnI8jeg+7zGhh r3/nehoX5EfkUhFIM6+BkpxU1JkFNF4= X-Envelope-To: erhard_f@mailbox.org X-Envelope-To: yuzhao@google.com X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: hannes@cmpxchg.org X-Envelope-To: nphamcs@gmail.com X-Envelope-To: senozhatsky@chromium.org X-Envelope-To: minchan@kernel.org X-Envelope-To: vbabka@kernel.org Message-ID: Date: Thu, 6 Jun 2024 10:49:14 +0800 MIME-Version: 1.0 Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc) Content-Language: en-US To: Yosry Ahmed , Erhard Furtner Cc: Yu Zhao , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Johannes Weiner , Nhat Pham , Sergey Senozhatsky , Minchan Kim , "Vlastimil Babka (SUSE)" References: <20240508202111.768b7a4d@yea> <20240515224524.1c8befbe@yea> <20240602200332.3e531ff1@yea> <20240604001304.5420284f@yea> <20240604134458.3ae4396a@yea> <20240604231019.18e2f373@yea> <20240606010431.2b33318c@yea> X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: C64CA1A0005 X-Rspam-User: X-Stat-Signature: 6xtmhofu4yue96c955t16hbau67r9hfi X-HE-Tag: 1717642185-251548 X-HE-Meta: U2FsdGVkX18adZyIbVVQyIBHq+GY/Rmmit/UL7IuGvDeb1wfTUlf4l1o1gq9GxAQn1Ke48JwhzmLFpR4rdPf01gL8q9QXGFcePICuObbVQaHKlSTX5G+kmYZhWFCOsGT4ErXhI7Ofacvv1W5JXvwFhl2XqmnBOJTv2xAsrc1mAl8lDaVBHNaotwx4aYYQ72hNRNcUUZk54U6g18HjQGM93aasd/YvNHuMIx3Ra3fBcP44eB2/4fr8XHwPEpf+fnhymkgfYJZwaXz6O7kYr9M8NLQ47w8diNSpRyP6KInI/tcIAQggdaCp2kp0xor8ELMOXMeDHxTxJiFk4rBHvTeAarZGb4Jyf6QF+EBOfjg7egdlJXVm0Y6Gpl98h7E+aQ+CkjLqS/whJgXrdLUVtJ7/dXGYCIBUX8CHNfKm8R3/ePcLXl+3kBb2NMkNa1CUZhCo2UrOqAsqz5f5CLjlMjHMA/IG0GuivEKCQCCGq+ODW/iulIJ35gyd46GdcVRAy8E/VrEEGd9VjHVguFpWtyYZC5iq1rrPhY4cAgpVaRyG224SGYW9JQFxoUEtabc5r33ntSVqldgUc54ZgE6pkvpt8XcoVDSRz2cyOIcTyGTskyU63dqx1pLq57iofVp0Iua0MMvyGvrKK9/szziRRFcF5OJT+Yx7ancUHjApTkBiTzHD9LtVXBB4Yu10t9MoeDzQmqxpnVo6vDL1fX1KxQGTG37jDfyGklHBrrK4ft7FA750A3z6fgnlNn/SovJg+elL2z03KgcCXO2GT+JbyjmcXPXDzztcW2hktTFdcNbk4qEG9kOwiHKhwF6vSemLbqJRhPhJFIH/rsdVLY6QytFITKhBhUVWCi+tuZnyI/caKKe4gTYEY0alK7f4LamUTgw7HsJ5edor9WsWMi9Dgkz1zMRJPE4V0BcxjJRKwx/UNLDU+aVQ7/KpSPakqXzeEFjQvTjGRhASwwvz++iJsb tSrPxBoj EqW1qTLbTNqj2vQ48r4UNe7uoU2FbgoNw84T4GsQRCqT8II1QdAU/H3l0eiivlnqtBTmbgRsCwB6NjQywB7F+tHyyEwZvarguCxu2Er3TJ2tHhfHi5/m8IsEcDTb2BqPkM/8NAJyC8Iu5+YNh2mdbExh4iUUdoLvSjbi11Mq9yCV29yU4pIztOlBfFNKf7SrENKf6bQV9MLWOPkFuwa2KaCFEPqB9Xy4w0y3icTHD6anRI8I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/6/6 07:41, Yosry Ahmed wrote: > On Wed, Jun 5, 2024 at 4:04 PM Erhard Furtner wrote: >> >> On Tue, 4 Jun 2024 20:03:27 -0700 >> Yosry Ahmed wrote: >> >>> Could you check if the attached patch helps? It basically changes the >>> number of zpools from 32 to min(32, nr_cpus). >> >> Thanks! The patch does not fix the issue but it helps. >> >> Means I still get to see the 'kswapd0: page allocation failure' in the dmesg, a 'stress-ng-vm: page allocation failure' later on, another kswapd0 error later on, etc. _but_ the machine keeps running the workload, stays usable via VNC and I get no hard crash any longer. >> >> Without patch kswapd0 error and hard crash (need to power-cycle) <3min. With patch several kswapd0 errors but running for 2 hrs now. I double checked this to be sure. > > Thanks for trying this out. This is interesting, so even two zpools is > too much fragmentation for your use case. > > I think there are multiple ways to go forward here: > (a) Make the number of zpools a config option, leave the default as > 32, but allow special use cases to set it to 1 or similar. This is > probably not preferable because it is not clear to users how to set > it, but the idea is that no one will have to set it except special use > cases such as Erhard's (who will want to set it to 1 in this case). > > (b) Make the number of zpools scale linearly with the number of CPUs. > Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this > approach is that with a large number of CPUs, too many zpools will > start having diminishing returns. Fragmentation will keep increasing, > while the scalability/concurrency gains will diminish. > > (c) Make the number of zpools scale logarithmically with the number of > CPUs. Maybe something like 4log2(nr_cpus). This will keep the number > of zpools from increasing too much and close to the status quo. The > problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) > will actually give a nr_zpools > nr_cpus. So we will need to come up > with a more fancy magic equation (e.g. 4log2(nr_cpus/4)). > > (d) Make the number of zpools scale linearly with memory. This makes > more sense than scaling with CPUs because increasing the number of > zpools increases fragmentation, so it makes sense to limit it by the > available memory. This is also more consistent with other magic > numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT). > > The problem is that unlike zswap trees, the zswap pool is not > connected to the swapfile size, so we don't have an indication for how > much memory will be in the zswap pool. We can scale the number of > zpools with the entire memory on the machine during boot, but this > seems like it would be difficult to figure out, and will not take into > consideration memory hotplugging and the zswap global limit changing. > > (e) A creative mix of the above. > > (f) Something else (probably simpler). > > I am personally leaning toward (c), but I want to hear the opinions of > other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else? > > In the long-term, I think we may want to address the lock contention > in zsmalloc itself instead of zswap spawning multiple zpools. > Agree, I think we should try to improve locking scalability of zsmalloc. I have some thoughts to share, no code or test data yet: 1. First, we can change the pool global lock to per-class lock, which is more fine-grained. 2. Actually, we only need to take per-zspage lock when malloc/free, only need to take class lock when its fullness changed. 3. If this is not enough, we can have fewer fullness groups, so the need to take class lock becomes fewer. (will need some test data) More comments are welcome. Thanks!