From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDAD1C433B4 for ; Thu, 1 Apr 2021 16:08:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 27A2960FE6 for ; Thu, 1 Apr 2021 16:08:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 27A2960FE6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 95B206B0075; Thu, 1 Apr 2021 12:08:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 931826B0078; Thu, 1 Apr 2021 12:08:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7AB216B007D; Thu, 1 Apr 2021 12:08:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0050.hostedemail.com [216.40.44.50]) by kanga.kvack.org (Postfix) with ESMTP id 5F9246B0075 for ; Thu, 1 Apr 2021 12:08:23 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1B0DD1803A82A for ; Thu, 1 Apr 2021 16:08:23 +0000 (UTC) X-FDA: 77984280486.24.10BFE93 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf14.hostedemail.com (Postfix) with ESMTP id B048DC0001FE for ; Thu, 1 Apr 2021 16:08:18 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id f2-20020a17090a4a82b02900c67bf8dc69so3332871pjh.1 for ; Thu, 01 Apr 2021 09:08:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fVS9owmVsNG+QULzYqIWFRouEzbiyw1Dm8jdaGD70hk=; b=bjdfSwVzdJJDEYHQ50CjgYRkcT2gSoprZBYt0araKfdDmyuLroMUizXCNYe8TZOqGG Fv2w9ZKWsPDnPiSfAI4vhwm3WcEcldkCtdXM0TCtNBhhZD3Nplzm9fYFR3bG8BaC766i fjHwEk4/PZSiSkF9jg9I2JyybsxGvpdHn4WuRsmFeA7UQ4cFg3P4rwIDquBo5brQ2avR X4/rXzvDiXj8gAoxLqUrmZ5K+M+iBMvvAa/Gv/63DfGIAkplBtOi1NpB/NX+B84T8NNB GSALGWdqMQH5AnJJU3RIs3/SNwiO6TfLWCZSuQ0czf2XktVFBByYJWG3ksw19QwTNlIz UtPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fVS9owmVsNG+QULzYqIWFRouEzbiyw1Dm8jdaGD70hk=; b=k37tNFWHpTXmQbMzrPVMyR7NtqEkD1RJ5OEi+NTRzeIOcoonZ6roypp3GG9ioCsjqq onYbLZDlBze2RSeFTaRCqzWzJUtcDlwWh9kHyjm3Q5lak/UqJr24hyOAFnO5w7QoBeUE dUWmfq9QGCvRhEmgsjF5SnKOKvWteF3HcsSmyWsPLlG14ABmsG+eth+3NGPfrSgrt9Xm UiKEPSkkjMF7TKEURYz3awS7VuJ+0qHQ9HOicODMHg+HTUOVmODJWPv2OyATFeJO+F7O S97648LTANnulLTsLvRwSQVOiyB0kKlbdNvvsJsUuDRRmhWjGLvyPdNT/Fk99J+D35oG CZOA== X-Gm-Message-State: AOAM5305m2GYPeZBQ+Fd9rdhB7zj7CjleJgabDv/+1Orwg5kUGRngmew S8F7cxhYuCHWRlgC7+cVA3BL8eedyVfCrTT4zSAYJQ== X-Google-Smtp-Source: ABdhPJwRNkFG9A2JstX8YeYmnZROu2KJe462Zd8Tu1Pfk/nVhS0+1NtmO6Sr4itP1jf/TAL1FEnj/edewwZGoAzxI/0= X-Received: by 2002:a17:90a:901:: with SMTP id n1mr9616339pjn.147.1617293299118; Thu, 01 Apr 2021 09:08:19 -0700 (PDT) MIME-Version: 1.0 References: <20210330101531.82752-1-songmuchun@bytedance.com> In-Reply-To: From: Muchun Song Date: Fri, 2 Apr 2021 00:07:42 +0800 Message-ID: Subject: Re: [External] Re: [RFC PATCH 00/15] Use obj_cgroup APIs to charge the LRU pages To: Johannes Weiner Cc: Roman Gushchin , Shakeel Butt , Greg Thelen , Michal Hocko , Andrew Morton , Vladimir Davydov , LKML , Linux MM , Xiongchun duan Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: B048DC0001FE X-Stat-Signature: 8gizfaa98ssszr5ok3ruexphuporqxis Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=mail-pj1-f45.google.com; client-ip=209.85.216.45 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617293298-291269 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Mar 31, 2021 at 11:17 PM Johannes Weiner wrote: > > On Tue, Mar 30, 2021 at 03:05:42PM -0700, Roman Gushchin wrote: > > On Tue, Mar 30, 2021 at 05:30:10PM -0400, Johannes Weiner wrote: > > > On Tue, Mar 30, 2021 at 11:58:31AM -0700, Roman Gushchin wrote: > > > > On Tue, Mar 30, 2021 at 11:34:11AM -0700, Shakeel Butt wrote: > > > > > On Tue, Mar 30, 2021 at 3:20 AM Muchun Song wrote: > > > > > > > > > > > > Since the following patchsets applied. All the kernel memory are charged > > > > > > with the new APIs of obj_cgroup. > > > > > > > > > > > > [v17,00/19] The new cgroup slab memory controller > > > > > > [v5,0/7] Use obj_cgroup APIs to charge kmem pages > > > > > > > > > > > > But user memory allocations (LRU pages) pinning memcgs for a long time - > > > > > > it exists at a larger scale and is causing recurring problems in the real > > > > > > world: page cache doesn't get reclaimed for a long time, or is used by the > > > > > > second, third, fourth, ... instance of the same job that was restarted into > > > > > > a new cgroup every time. Unreclaimable dying cgroups pile up, waste memory, > > > > > > and make page reclaim very inefficient. > > > > > > > > > > > > We can convert LRU pages and most other raw memcg pins to the objcg direction > > > > > > to fix this problem, and then the LRU pages will not pin the memcgs. > > > > > > > > > > > > This patchset aims to make the LRU pages to drop the reference to memory > > > > > > cgroup by using the APIs of obj_cgroup. Finally, we can see that the number > > > > > > of the dying cgroups will not increase if we run the following test script. > > > > > > > > > > > > ```bash > > > > > > #!/bin/bash > > > > > > > > > > > > cat /proc/cgroups | grep memory > > > > > > > > > > > > cd /sys/fs/cgroup/memory > > > > > > > > > > > > for i in range{1..500} > > > > > > do > > > > > > mkdir test > > > > > > echo $$ > test/cgroup.procs > > > > > > sleep 60 & > > > > > > echo $$ > cgroup.procs > > > > > > echo `cat test/cgroup.procs` > cgroup.procs > > > > > > rmdir test > > > > > > done > > > > > > > > > > > > cat /proc/cgroups | grep memory > > > > > > ``` > > > > > > > > > > > > Patch 1 aims to fix page charging in page replacement. > > > > > > Patch 2-5 are code cleanup and simplification. > > > > > > Patch 6-15 convert LRU pages pin to the objcg direction. > > > > > > > > > > The main concern I have with *just* reparenting LRU pages is that for > > > > > the long running systems, the root memcg will become a dumping ground. > > > > > In addition a job running multiple times on a machine will see > > > > > inconsistent memory usage if it re-accesses the file pages which were > > > > > reparented to the root memcg. > > > > > > > > I agree, but also the reparenting is not the perfect thing in a combination > > > > with any memory protections (e.g. memory.low). > > > > > > > > Imagine the following configuration: > > > > workload.slice > > > > - workload_gen_1.service memory.min = 30G > > > > - workload_gen_2.service memory.min = 30G > > > > - workload_gen_3.service memory.min = 30G > > > > ... > > > > > > > > Parent cgroup and several generations of the child cgroup, protected by a memory.low. > > > > Once the memory is getting reparented, it's not protected anymore. > > > > > > That doesn't sound right. > > > > > > A deleted cgroup today exerts no control over its abandoned > > > pages. css_reset() will blow out any control settings. > > > > I know. Currently it works in the following way: once cgroup gen_1 is deleted, > > it's memory is not protected anymore, so eventually it's getting evicted and > > re-faulted as gen_2 (or gen_N) memory. Muchun's patchset doesn't change this, > > of course. But long-term we likely wanna re-charge such pages to new cgroups > > and avoid unnecessary evictions and re-faults. Switching to obj_cgroups doesn't > > help and likely will complicate this change. So I'm a bit skeptical here. > > We should be careful with the long-term plans. > > The zombie issue is a pretty urgent concern that has caused several > production emergencies now. It needs a fix sooner rather than later. Thank you very much for clarifying the problem for me. I do agree with you. This issue should be fixed ASAP. Using objcg is a good choice. Dying objcg should not be a problem. Because the size of objcg is so small compared to memcg. Thanks. > > The long-term plans of how to handle shared/reused data better will > require some time to work out. There are MANY open questions around > recharging to arbitrary foreign cgroup users. Like how to identify > accesses after the page's cgroup has been deleted: Do we need to > annotate every page cache lookup? Do we need to inject minor faults to > recharge mmapped pages? We can't wait for this discussion to complete. > > I also think the objcg is helping with that direction rather than > getting in the way because: > > - The old charge moving code we have for LRU pages isn't reusable > anyway. It relies on optimistic locking to work, and may leave > memory behind in arbitrary and unpredictable ways. After a few > cycles, objects tend to be spread all over the place. > > The objcg provides a new synchronization scheme that will always > work because the readside (page/object to memcg lookup) needs to be > prepared for the memcg to change and/or die at any time. > > - There isn't much point in recharging only some of the abandoned > memory. We've tried per-allocation class reparenting and it didn't > work out too well. Newly accounted allocations crop up faster than > we can conjure up tailor-made reparenting schemes for them. > > The objcg provides a generic reference and reassignment scheme that > can be used by all tracked objects. > > Importantly, once we have a central thing as LRU pages converted, we > can require all new allocation tracking to use objcg from the start. > > > Also, in my experience the pagecache is not the main/worst memcg reference > > holder (writeback structures are way worse). Pages are relatively large > > (in comparison to some slab objects), and rarely there is only one page pinning > > a separate memcg. > > I've seen that exact thing cause zombies to pile up: one or two pages > in the old group, pinned by the next instance of a job. If the job has > a sufficiently large working set, this can pin a LOT of dead > cgroups. Is it the biggest or most problematic source of buildups? > Maybe not. But there is definitely cause to fix it. > > LRU pages are also the most difficult to convert because of their rich > interactions. It's a good test of the api. If it works for those > pages, converting everybody else will be much simpler. > > And again, as the core memory consumer it sets the tone of how memcg > rmapping is supposed to work for new and existing object types. This > helps align ongoing development. > > > And switching to obj_cgroup doesn't completely eliminate the > > problem: we just switch from accumulating larger mem_cgroups to > > accumulating smaller obj_cgroups. > > In your own words, the discrepancy between tiny objects pinning large > memcgs is a problem. objcgs are smaller than most objects, so it's not > much different as if an object were simply a few bytes bigger in size. > A memcg on the other hand is vastly bigger than most objects. It's > also composed of many allocations and so causes more fragmentation. > > Another issue is that memcgs with abandoned objects need to be visited > by the reclaimer on every single reclaim walk, a hot path. The list of > objcgs on the other hand is only iterated when the cgroup is deleted, > which is not a fast path. It's also rare that parents with many dead > children are deleted at all (system.slice, workload.slice etc.) > > So no, I would say for all intents and purposes, it fixes all the > problems we're having with zombie memcgs. > > > With all this said, I'm not necessarily opposing the patchset, but it's > > necessary to discuss how it fits into the long-term picture. > > E.g. if we're going to use obj_cgroup API for page-sized objects, shouldn't > > we split it back into the reparenting and bytes-sized accounting parts, > > as I initially suggested. And shouldn't we move the reparenting part to > > the cgroup core level, so we could use it for other controllers > > (e.g. to fix the writeback problem). > > Yes, I do think we want to generalize it. But I wouldn't say it's a > requirement for these patches, either: > > - The byte-sized accounting part is one atomic_t. That's 4 bytes > pinned unnecessarily - compared to an entire struct memcg right now > which includes memory accounting, swap accounting, byte accounting, > and a whole lot of other things likely not used by the stale object. > > - The cgroup generalization is a good idea too, but that doesn't > really change the callsites either. Unless you were thinking of > renaming, but objcg seems like a good, generic fit for a name to > describe the linkage between objects to a cgroup. > > The memcg member will need to change into something generic (a > css_set type mapping), but that can likely be hidden behind > page_memcg(), objcg_memcg() and similar helpers. > > Both of these aspects can be improved incrementally.