From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9ED2AC433F5 for ; Wed, 12 Jan 2022 04:48:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D1B866B011C; Tue, 11 Jan 2022 23:48:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CCB1E6B011D; Tue, 11 Jan 2022 23:48:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6C356B011E; Tue, 11 Jan 2022 23:48:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0116.hostedemail.com [216.40.44.116]) by kanga.kvack.org (Postfix) with ESMTP id A71546B011C for ; Tue, 11 Jan 2022 23:48:40 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 33E2D8122D96 for ; Wed, 12 Jan 2022 04:48:40 +0000 (UTC) X-FDA: 79020404400.03.97C5910 Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) by imf29.hostedemail.com (Postfix) with ESMTP id 0478012000D for ; Wed, 12 Jan 2022 04:48:37 +0000 (UTC) Received: by mail-yb1-f176.google.com with SMTP id g14so3001266ybs.8 for ; Tue, 11 Jan 2022 20:48:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zLZAC3CMSCIXpnQURW0tWmpIOKmVcmIrl+Z52NxTeuQ=; b=LCIkAOvJVwWFouI+xP+uISMFfHNgU9FxU3p7BFxfiPXa+M0vBPwjMymrSUDgziWttR p144ZmbB3+yXZm0rpBTHosOYbFVunyGfkCsGwtkDPSkwJwQnfeXAKLxiob6mpjH/1qnr ZFIvxkOPQh3H2G2HUVdqapmUPxUsyAJcsLEYGgC1KBGfLGl0gmadY8XUorqu6xa5fihw xN5juk8uCbIX47eUEFIAhfe+y9Idl6gNJlzF3gnriES26RuyD3SYMZS281eEmMpgUspK l72b5hAJplFcyjP5xYy4kh62C4EQIjBpkW+ileNPV9Ak7yAp3Yp+Bxpawtqe2zIBZYoE N3mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zLZAC3CMSCIXpnQURW0tWmpIOKmVcmIrl+Z52NxTeuQ=; b=69J6T51xxYL4ORw31QOVidpx9CLzBsy6uny+o7aSxUzQLSSRvoJ6p6F1qLadcjwmKv +WgzlsNGEYzJclbLd+xbHHd9Vp8Ec4djDhItb+PUaBCbUkaLX1yY2VAJ+t61oMZ9V1hH IyeVqiB/6Zb/u/mDGqcDfK5ujnIGQr0nf0lZARNohSd1kb5tXf1F20KBxciiv3uDN6Vx x+liPLNLenGqgYPgDvVCjJmwipmqs1L0XBUZntPnTbK3Qo6KB4Og6IjUNaao8AMG2KEE osIPKWM+seNmikWSejvsk3P4ldmWjeA6R5ufTt8U9QUFkaiM9ai3k4L2DpFimI4z0ysr rUAg== X-Gm-Message-State: AOAM532Fn30TsetBAcpNMFYV2OKDwavMcavSjiAEGz5RrjruFRQYB6Ie 0LvS5YKEm2sSl2/lKIv8hNIDjBU3lrmIWG9RXVoa7A== X-Google-Smtp-Source: ABdhPJzM6r3HV7zixUCi+df2G19UeRviY+iC+6vrvYNbRVy+qhG00MN449rGwCohBbkIGf44mBkrBtf62rcRCnOCTUs= X-Received: by 2002:a25:abcb:: with SMTP id v69mr10868130ybi.317.1641962916903; Tue, 11 Jan 2022 20:48:36 -0800 (PST) MIME-Version: 1.0 References: <20211220085649.8196-1-songmuchun@bytedance.com> <20211220085649.8196-11-songmuchun@bytedance.com> In-Reply-To: From: Muchun Song Date: Wed, 12 Jan 2022 12:48:00 +0800 Message-ID: Subject: Re: [PATCH v5 10/16] mm: list_lru: allocate list_lru_one only when needed To: Roman Gushchin Cc: Matthew Wilcox , Andrew Morton , Johannes Weiner , Michal Hocko , Vladimir Davydov , Shakeel Butt , Yang Shi , Alex Shi , Wei Yang , Dave Chinner , trond.myklebust@hammerspace.com, anna.schumaker@netapp.com, jaegeuk@kernel.org, chao@kernel.org, Kari Argillander , linux-fsdevel , LKML , Linux Memory Management List , linux-nfs@vger.kernel.org, Qi Zheng , Xiongchun duan , Fam Zheng , Muchun Song Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 0478012000D X-Stat-Signature: e3eu18gqtgmarj65t5msc7bt6bp6sft9 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=LCIkAOvJ; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf29.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.219.176 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam08 X-HE-Tag: 1641962917-192498 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jan 12, 2022 at 4:00 AM Roman Gushchin wrote: > > On Mon, Dec 20, 2021 at 04:56:43PM +0800, Muchun Song wrote: > > In our server, we found a suspected memory leak problem. The kmalloc-32 > > consumes more than 6GB of memory. Other kmem_caches consume less than > > 2GB memory. > > > > After our in-depth analysis, the memory consumption of kmalloc-32 slab > > cache is the cause of list_lru_one allocation. > > > > crash> p memcg_nr_cache_ids > > memcg_nr_cache_ids = $2 = 24574 > > > > memcg_nr_cache_ids is very large and memory consumption of each list_lru > > can be calculated with the following formula. > > > > num_numa_node * memcg_nr_cache_ids * 32 (kmalloc-32) > > > > There are 4 numa nodes in our system, so each list_lru consumes ~3MB. > > > > crash> list super_blocks | wc -l > > 952 > > > > Every mount will register 2 list lrus, one is for inode, another is for > > dentry. There are 952 super_blocks. So the total memory is 952 * 2 * 3 > > MB (~5.6GB). But the number of memory cgroup is less than 500. So I > > guess more than 12286 containers have been deployed on this machine (I > > do not know why there are so many containers, it may be a user's bug or > > the user really want to do that). And memcg_nr_cache_ids has not been > > reduced to a suitable value. This can waste a lot of memory. > > But on the other side you increase the size of struct list_lru_per_memcg, > so if number of cgroups is close to memcg_nr_cache_ids, we can actually > waste more memory. The saving comes from the fact that we currently allocate scope for every memcg to be able to be tracked on every superblock instantiated in the system, regardless of whether that superblock is even accessible to that memcg. In theory, increasing struct list_lru_per_memcg is not significant, most savings is from decreasing the number of allocations of struct list_lru_per_memcg. > I'm not saying the change is not worth it, but would be > nice to add some real-world numbers. OK. I will do a test. > > Or it's all irrelevant and is done as a preparation to the conversion to xarray? Right. It's also a preparation to transfer to xarray. > If so, please, make it clear. Will do. Thanks.