From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1521C433F5 for ; Wed, 15 Dec 2021 12:10:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 62DF76B0073; Wed, 15 Dec 2021 07:09:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5DDAF6B0074; Wed, 15 Dec 2021 07:09:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A5786B0075; Wed, 15 Dec 2021 07:09:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0240.hostedemail.com [216.40.44.240]) by kanga.kvack.org (Postfix) with ESMTP id 3D54E6B0073 for ; Wed, 15 Dec 2021 07:09:58 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 0C42182FF8 for ; Wed, 15 Dec 2021 12:09:48 +0000 (UTC) X-FDA: 78919909656.20.002DD0E Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) by imf30.hostedemail.com (Postfix) with ESMTP id 119718000F for ; Wed, 15 Dec 2021 12:09:46 +0000 (UTC) Received: by mail-yb1-f176.google.com with SMTP id y68so54615219ybe.1 for ; Wed, 15 Dec 2021 04:09:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=medD3Xi3kZKH5q7bagszydyhriMt4qJw4H1Wjw4BIeA=; b=hFy0KgQxd2bJci/Xd+n8sv+Gu5RmYMvEDhUBuj2Yrq8ctOAyB8hlUNmxpM2IiWLKGy GE3vvn8E0z4jskkFd9srdqyZ83G3qGSVOcrD2grWx5AMKa7YPQ48/Vv0KqUhJi54LfOy dD7WRE2C4pHHS/JIsZt35udpFpfLB/bfGhx9yBgUoW8bTi5myNlDaZXqEKI6Kx46FisJ 68kZKS3svXvUdYbK5/j6C34dFY+I1Rbbh4hEBrVj5Z8GkSaIoGT6INxX0WmFow5netem tEe/Dm5Fek5uzcugw8IodwEebZjX23z7LIjzHm5KAIPoJ2dg3exHdgGp4uP3a5We1NHj TTAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=medD3Xi3kZKH5q7bagszydyhriMt4qJw4H1Wjw4BIeA=; b=no4G1OR8m10Me3CYGmWdZkayeQCAynibr3+vOhA+rXzqdBcbW+FFNOIcKsJDmele4A 7wHhOH9On9moV8wv63WzO3N7PW/IP6/LWWRFEDdlrogAVbJ8LY9HJTDFVbhzwQshedZ9 zIaZDxeXFbXrAosq2W1AUXprQG9j/KoWTQVd9UVNRtntd9tfxc+q6NYx1LnVczrD6hXm ihXaVQVU2OLvKa/3WUDkqYTxNnmldypYju2xVzdJpxPlLW5+u2xi+Y6mvy1IWravdeWM 7wjq1wbV5KiNEgk7YU2eR9EscUtwIFyVx/XOHirT7+m+BBGgNsxPHm4vmI4kctcOvVQG 5zlQ== X-Gm-Message-State: AOAM530sqoJgqssExKHoV4LnS1G5p6NxGYesbADDUYmdQRQs9hxHYcUJ 3ru3jYFRQCE++P3XX3RJWENtz/VpJdG1n2CF14aZbg== X-Google-Smtp-Source: ABdhPJwQ0At3Bjn2QQCan5WHYT82Z/7G6PUgdH5xVAc30+lYih6V821vw+gn8W1jOKI1DU2w2zfhyVDYOnqoiKKOUGU= X-Received: by 2002:a05:6902:52b:: with SMTP id y11mr5529088ybs.199.1639570186129; Wed, 15 Dec 2021 04:09:46 -0800 (PST) MIME-Version: 1.0 References: <20211213165342.74704-1-songmuchun@bytedance.com> <20211213165342.74704-2-songmuchun@bytedance.com> In-Reply-To: From: Muchun Song Date: Wed, 15 Dec 2021 20:09:10 +0800 Message-ID: Subject: Re: [PATCH v4 01/17] mm: list_lru: optimize memory consumption of arrays of per cgroup lists To: Johannes Weiner Cc: Matthew Wilcox , Andrew Morton , Michal Hocko , Vladimir Davydov , Shakeel Butt , Roman Gushchin , Yang Shi , Alex Shi , Wei Yang , Dave Chinner , trond.myklebust@hammerspace.com, anna.schumaker@netapp.com, jaegeuk@kernel.org, chao@kernel.org, Kari Argillander , linux-fsdevel , LKML , Linux Memory Management List , linux-nfs@vger.kernel.org, Qi Zheng , Xiongchun duan , fam.zheng@bytedance.com, Muchun Song Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=hFy0KgQx; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.219.176 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 119718000F X-Stat-Signature: 5y9mggbwj4s8eee41o3kgeyd65x7q1su X-HE-Tag: 1639570186-91563 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Dec 14, 2021 at 9:38 PM Johannes Weiner wrote: > > On Tue, Dec 14, 2021 at 12:53:26AM +0800, Muchun Song wrote: > > The list_lru uses an array (list_lru_memcg->lru) to store pointers > > which point to the list_lru_one. And the array is per memcg per node. > > Therefore, the size of the arrays will be 10K * number_of_node * 8 ( > > a pointer size on 64 bits system) when we run 10k containers in the > > system. The memory consumption of the arrays becomes significant. The > > more numa node, the more memory it consumes. > > The complexity for the lists themselves is still nrmemcgs * nrnodes > right? But the rcu_head goes from that to nrmemcgs. Right. > > > I have done a simple test, which creates 10K memcg and mount point > > each in a two-node system. The memory consumption of the list_lru > > will be 24464MB. After converting the array from per memcg per node > > to per memcg, the memory consumption is going to be 21957MB. It is > > reduces by 2.5GB. In our AMD servers, there are 8 numa nodes in > > those system, the memory consumption could be more significant. > > The code looks good to me, but it would be useful to include a > high-level overview of the new scheme, explain that the savings come > from the rcu heads, that it simplifies the alloc/dealloc path etc. Will do in the next version. > > With that, > > > Signed-off-by: Muchun Song Thanks. > > Acked-by: Johannes Weiner