From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C9DEC25B75 for ; Thu, 6 Jun 2024 04:47:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7776C6B009D; Thu, 6 Jun 2024 00:47:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 727466B009E; Thu, 6 Jun 2024 00:47:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 615706B009F; Thu, 6 Jun 2024 00:47:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4373B6B009D for ; Thu, 6 Jun 2024 00:47:14 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AE555140F7B for ; Thu, 6 Jun 2024 04:47:13 +0000 (UTC) X-FDA: 82199229546.08.D0C8D2C Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) by imf04.hostedemail.com (Postfix) with ESMTP id A105340009 for ; Thu, 6 Jun 2024 04:47:09 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=extDQOVZ; spf=pass (imf04.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717649231; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4nLB6EbYrbegnA9wITWnbt4aOb9qcGFKJTII1fSGomQ=; b=GRFeNuf5VM4lmRMD4pqhXiOlRB3RZ9ATnO0B7JFIZ5bmFOvMUBLeEs/JqiWKA+R41EgYji 890LoaiHdOoC0BFCzTf+EMyD9JXldHl+sDw4Q2Nw/zU+XCjpRjIQH77qVKnk6fEae90rBQ BwmfJWiL5iB2qrs+Ot0DwBybBnrVgNE= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=extDQOVZ; spf=pass (imf04.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717649231; a=rsa-sha256; cv=none; b=UaBTkkNHTwCRnknEhQNy5p0UPPXZz31IxNt298pNFIGE5+0J4z0y2+g4HkO1sl2ORyOxsG DHkdFaKKQBp0qSazgdOQm7FT3t4zr5RaBV0K3wXN783YzCQwJNfj9+BCfHSy67KULyi/rL +vMI4x2ZLzNu7K/bJb6oMdZhOAFB7Fk= X-Envelope-To: senozhatsky@chromium.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1717649225; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4nLB6EbYrbegnA9wITWnbt4aOb9qcGFKJTII1fSGomQ=; b=extDQOVZpl/9ARNAyQKxODmd+M/hxCklhwtcl03lSAtLo3ObGUxnuJu4xHv3SSORnPVTuR D1J+k3cp5F5p3tT/nL1cT4al7bC5E174jbetvR3Qxjs72hGS1q983xDoLCI2v9A1yzKoG2 H0Fq2bYQG0OTrdJUvHrcs6TbdEoBw8A= X-Envelope-To: yosryahmed@google.com X-Envelope-To: erhard_f@mailbox.org X-Envelope-To: yuzhao@google.com X-Envelope-To: linux-mm@kvack.org X-Envelope-To: linux-kernel@vger.kernel.org X-Envelope-To: linuxppc-dev@lists.ozlabs.org X-Envelope-To: hannes@cmpxchg.org X-Envelope-To: nphamcs@gmail.com X-Envelope-To: minchan@kernel.org X-Envelope-To: vbabka@kernel.org Message-ID: <6335c05d-9493-4b03-85a7-f2dd91db9451@linux.dev> Date: Thu, 6 Jun 2024 12:46:36 +0800 MIME-Version: 1.0 Subject: Re: kswapd0: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0 (Kernel v6.5.9, 32bit ppc) Content-Language: en-US To: Sergey Senozhatsky Cc: Yosry Ahmed , Erhard Furtner , Yu Zhao , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Johannes Weiner , Nhat Pham , Minchan Kim , "Vlastimil Babka (SUSE)" References: <20240604231019.18e2f373@yea> <20240606010431.2b33318c@yea> <20240606043156.GC11718@google.com> X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou In-Reply-To: <20240606043156.GC11718@google.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_OUT X-Stat-Signature: cm1hzbgirfrxeco18e6o869y3iokr4wy X-Rspamd-Queue-Id: A105340009 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1717649229-71115 X-HE-Meta: U2FsdGVkX1+QOAea2HK1Ao3Spp7BDHeqgIht+lP2MTWyso75i8rhlFIBCj5f2RG8bqg5hdxF/accq5hJ1Whvs81qCNvikYx8KL5Mwz6Tt9nUDSjazy2LSPgjp+V6lGJ6d4jLVljdSIypFIsWHdgFzBr3/OhJ0qqxC/pYfVyzeV4O+1IQ4n+fQlyovyg7WdB3po9661iaBDBhyXpytfb9jzLx/+fd+5iMiKGuOgUd1xKGsRmLm5R+NW7ftR/tRbrwrGFMRTBDdLVyii8GrvdL1oBE/8legpRY5m5zxobDE7rkD9o4MDyMP02tGhM0hITcU2OMpPuYH4y0Z9D+je8Bw8Xs7421X0NVLlnokndrq8MGg1pM5/WVmym1xfApPmHn/EpDf3KzcZ7MLXzRMIWn5rWL7Urc5mGvZGundw2VXufkqiiRJ4S2L4hvbM1mv0EFYsYCvSU3Rj42yE4gSQfuoQ1W5Z7dJmEpHIK+lHgWqZi59podIiqVDWMlBR/WThbz2rDFWIixtoA6CJoH+seZIF5y0XY4Srm3/RyCcGWL0mhmNItJNVgnEQ5eAERkYo/UyX9RMOqKL5o6+8O7HvdAG/4xnmJGa2oFRdozHYc/ANcRMdFO4aiwrmYCEa0LpBHTNA6QzvB85Xer9bc27GJAWCozYfg9We5S5uhmK8ZycjRA/JtZ6ttNNImMpLp0F6g03lErAV60kjFPZ97Qn4cwGqpBEziEt6m8pdiX70P1ZrtrdaX/KI7C00U1SWnGAmM+0/cLvwcIdpR5WfPVOtU5sV5UfGCoksxcHYW6Zvj6bTIjQjJia+cMQMX1EokTKPdHf2FaewXaVJrkJLpRy77ZpkSlOiY592vfTDUelapNqoOWDwRt+jhkX+iy5Ceba0JcoHxaYX1Xek8cnpaa1TRd6ODB3vAIbyLeQWhrTUhhzD4xpCwWmI//oI/B2d8NyuByhvnlCRKlM+IvPP+XsPq lSjA9hiM UoYq3hdXfFWQGSWe4zzBlsQBInzmLdSkjfCYyE7lDM3JLiZruYebeUNxh9ArxALbADERUo5pl/1PwYTJBYSKLUh43UJZ7aQHwmPXuHvATzSePBab+CYe9SNb8dhu5dj7L4jmlrX3/FGmjffq+dLny8LQDk1juceBgNTsEOFkJAeRwphqkor9HBj/UQIUf/kJUtk+W+7rrxGuxk2n7BBw6DXvoqvmoq1cbrZbk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/6/6 12:31, Sergey Senozhatsky wrote: > On (24/06/06 10:49), Chengming Zhou wrote: >>> Thanks for trying this out. This is interesting, so even two zpools is >>> too much fragmentation for your use case. >>> >>> I think there are multiple ways to go forward here: >>> (a) Make the number of zpools a config option, leave the default as >>> 32, but allow special use cases to set it to 1 or similar. This is >>> probably not preferable because it is not clear to users how to set >>> it, but the idea is that no one will have to set it except special use >>> cases such as Erhard's (who will want to set it to 1 in this case). >>> >>> (b) Make the number of zpools scale linearly with the number of CPUs. >>> Maybe something like nr_cpus/4 or nr_cpus/8. The problem with this >>> approach is that with a large number of CPUs, too many zpools will >>> start having diminishing returns. Fragmentation will keep increasing, >>> while the scalability/concurrency gains will diminish. >>> >>> (c) Make the number of zpools scale logarithmically with the number of >>> CPUs. Maybe something like 4log2(nr_cpus). This will keep the number >>> of zpools from increasing too much and close to the status quo. The >>> problem is that at a small number of CPUs (e.g. 2), 4log2(nr_cpus) >>> will actually give a nr_zpools > nr_cpus. So we will need to come up >>> with a more fancy magic equation (e.g. 4log2(nr_cpus/4)). >>> >>> (d) Make the number of zpools scale linearly with memory. This makes >>> more sense than scaling with CPUs because increasing the number of >>> zpools increases fragmentation, so it makes sense to limit it by the >>> available memory. This is also more consistent with other magic >>> numbers we have (e.g. SWAP_ADDRESS_SPACE_SHIFT). >>> >>> The problem is that unlike zswap trees, the zswap pool is not >>> connected to the swapfile size, so we don't have an indication for how >>> much memory will be in the zswap pool. We can scale the number of >>> zpools with the entire memory on the machine during boot, but this >>> seems like it would be difficult to figure out, and will not take into >>> consideration memory hotplugging and the zswap global limit changing. >>> >>> (e) A creative mix of the above. >>> >>> (f) Something else (probably simpler). >>> >>> I am personally leaning toward (c), but I want to hear the opinions of >>> other people here. Yu, Vlastimil, Johannes, Nhat? Anyone else? >>> >>> In the long-term, I think we may want to address the lock contention >>> in zsmalloc itself instead of zswap spawning multiple zpools. > > Sorry, I'm sure I'm not following this discussion closely enough, > has the lock contention been demonstrated/proved somehow? lock-stats? Yosry has some stats in his commit b8cf32dc6e8c ("mm: zswap: multiple zpools support"), and I have also seen some locking contention when using zram to test kernel building, since zram still has only one pool. > >> Agree, I think we should try to improve locking scalability of zsmalloc. >> I have some thoughts to share, no code or test data yet: >> >> 1. First, we can change the pool global lock to per-class lock, which >> is more fine-grained. > > Commit c0547d0b6a4b6 "zsmalloc: consolidate zs_pool's migrate_lock > and size_class's locks" [1] claimed no significant difference > between class->lock and pool->lock. Ok, I haven't looked into the history much, that seems preparation of trying to introduce reclaim in the zsmalloc? Not sure. But now with the reclaim code in zsmalloc has gone, should we change back to the per-class lock? Which is obviously more fine-grained than the pool lock. Actually, I have just done it, will test to get some data later. Thanks. > > [1] https://lkml.kernel.org/r/20221128191616.1261026-4-nphamcs@gmail.com