From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94AAFC43460 for ; Wed, 7 Apr 2021 15:42:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3096561382 for ; Wed, 7 Apr 2021 15:42:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3096561382 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B639D6B0036; Wed, 7 Apr 2021 11:42:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B007B6B0072; Wed, 7 Apr 2021 11:42:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A0BF6B007D; Wed, 7 Apr 2021 11:42:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0086.hostedemail.com [216.40.44.86]) by kanga.kvack.org (Postfix) with ESMTP id 8207B6B0036 for ; Wed, 7 Apr 2021 11:42:46 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 3978E8248D52 for ; Wed, 7 Apr 2021 15:42:46 +0000 (UTC) X-FDA: 78005988732.04.7DFE62D Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) by imf19.hostedemail.com (Postfix) with ESMTP id A125790009F1 for ; Wed, 7 Apr 2021 15:42:37 +0000 (UTC) Received: by mail-lj1-f170.google.com with SMTP id u20so21199900lja.13 for ; Wed, 07 Apr 2021 08:42:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=bk1tr8gnipD6uFCQutltHcbiRHj4dh2dyGytQA9wCWU=; b=eALrPYdSxG5ztUUc6HNLmohxDavZmiWNBJfKHWtxO2z25tbLT0s77VUc8jnKkT7fTG Jy77kKUVud/7YniQoUals7Kr1C7oq9aQSAzuWi/Wg4BChaO5rokhNH8FUyH4cBoqSXwt Og8djPFUGKELXZauXW7sX5lIy0mvXskCnW3BmQBwZr5Xy3+xnn1kCFzCspcZNg+15DIL N0TUbrS0DZ/SCy9hkCIFrwB6o8AsZzFPlj6AkxEbN0xSYGh7XXaKEGndh2XJJOTqfHrl 5YYZEu6C8eKvN5UdN8CDz4w0tBd9Vi+mLclahiyl2yTQb3Jlf4yQH/AI8AjXwleRiWXP vLOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=bk1tr8gnipD6uFCQutltHcbiRHj4dh2dyGytQA9wCWU=; b=qUOwZgW0P/TZZ2Vqp8kWmaEvIc+xkIg1PpQcDHD9xdaPUF1GEYxulvhDIEgC1zTC2W 8PWIbyaqvOkw5mdqH6mzhMQnt0EPvie/WLsxF1q9OnYPMf3jpDe0WSjNeh4bj1TevKnH HIUUijlT48JuGg5b3T5T3W6L4ct8g1S+ng7wvRGRhJPkrTf6yhAFbFjkF2uTiMzNdQQv McsIxmGH5G29UqUolt3+MXffkiy1VDmWPi2k7tjwpdyTtFkD0pfrEYAl2P5d/5nq1kLj ryzdmSd2Y2Ddz4AphvCVqZnHlcleFlUGEuKauohY3fMIxv/PJ1/NiDYGavIPedVmGGFF 1c/g== X-Gm-Message-State: AOAM533nW9OtfCf6o4lbYOHysbBuoN7SXeK5R8dnctGRSBR5tuXeTolZ sVWAGkc96DsCjD4xODbJN7Lztf1Td69rsE43rMOLLg== X-Google-Smtp-Source: ABdhPJyItEPMRdkdL43c6GKpBK8KvHU+D+sruSGo8iOAoXefA95/2e3w/6+Yu9MghbdpFLXPZb1TTFuchX6pCiMoocY= X-Received: by 2002:a2e:85d9:: with SMTP id h25mr2522397ljj.81.1617810163862; Wed, 07 Apr 2021 08:42:43 -0700 (PDT) MIME-Version: 1.0 References: <20210405054848.GA1077931@in.ibm.com> In-Reply-To: From: Shakeel Butt Date: Wed, 7 Apr 2021 08:42:31 -0700 Message-ID: Subject: Re: High kmalloc-32 slab cache consumption with 10k containers To: Michal Hocko Cc: Bharata B Rao , Andrew Morton , LKML , Linux MM , linux-fsdevel , "Aneesh Kumar K.V" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A125790009F1 X-Stat-Signature: wdxfz3ncju6bwebarh8ru81rbijzepuc Received-SPF: none (google.com>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=mail-lj1-f170.google.com; client-ip=209.85.208.170 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617810157-602965 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Apr 7, 2021 at 4:55 AM Michal Hocko wrote: > > On Mon 05-04-21 11:18:48, Bharata B Rao wrote: > > Hi, > > > > When running 10000 (more-or-less-empty-)containers on a bare-metal Power9 > > server(160 CPUs, 2 NUMA nodes, 256G memory), it is seen that memory > > consumption increases quite a lot (around 172G) when the containers are > > running. Most of it comes from slab (149G) and within slab, the majority of > > it comes from kmalloc-32 cache (102G) > > Is this 10k cgroups a testing enviroment or does anybody really use that > in production? I would be really curious to hear how that behaves when > those containers are not idle. E.g. global memory reclaim iterating over > 10k memcgs will likely be very visible. I do remember playing with > similar setups few years back and the overhead was very high. > -- I can tell about our environment. Couple of thousands of memcgs (~2k) are very normal on our machines as machines can be running 100+ jobs (and each job can manage their own sub-memcgs). However each job can have a high number of mounts. There is no local disk and each package of the job is remotely mounted (a bit more complicated). We do have issues with global memory reclaim but mostly the proactive reclaim makes the global reclaim a tail issue (and at tail it often does create havoc).