From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 662E2C43334 for ; Tue, 28 Jun 2022 01:31:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB7D78E0002; Mon, 27 Jun 2022 21:31:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C67C98E0001; Mon, 27 Jun 2022 21:31:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2FA78E0002; Mon, 27 Jun 2022 21:31:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A32EC8E0001 for ; Mon, 27 Jun 2022 21:31:52 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 73D99211F7 for ; Tue, 28 Jun 2022 01:31:52 +0000 (UTC) X-FDA: 79625918064.17.CE8BED6 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by imf02.hostedemail.com (Postfix) with ESMTP id 17A8780032 for ; Tue, 28 Jun 2022 01:31:51 +0000 (UTC) Received: by mail-wr1-f41.google.com with SMTP id n1so15400607wrg.12 for ; Mon, 27 Jun 2022 18:31:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=C1kij541w9Z+pa/MZJLadURpUoXi65YxPa2CRYu92jA=; b=cKi7cvajC+uPnxXFRitvLXro1PUT/Wjn852N+RTtDGloE02o9QdgJTYFJ60FUFMovm wDsovPIwtXkzKxEIX6RWu4yqHyd8RsV0AbMhT6jVO466kcqb0qJT0WYwxTGOkWv2Dzla CgEaWChHuOju356H0vVFjvdfq4RgLDZzZV+0k7TP+bcN18+PdnbOpAWG4q1k195uK//k wMv+8Y+jFg53RICi3MXHPjS3igvGyojvMC9H6/XYnjgNefFJmgeN2DqQlsqCQtGw9yGf fIaDBMVILyUZ2mjGMjmX4zH7caCmokzf88MKEiqlKpvYEWCiOXS4yDcjVnikX/tf2cbI IDWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=C1kij541w9Z+pa/MZJLadURpUoXi65YxPa2CRYu92jA=; b=Een1eSuCv+rlilnV1G44HQ3ZrdfgfFb8ToP/O2SesT6P5kdTEqrh6956YOe2GplWLL MxRL6zlXPA8DfLHnpfFJEgwHXsPhOOciPZwP1+aRAwAQRwLiYKrYfpVfoIL4yJ5t9hUl cTWwHDqLMrGtileA+H3+TYeVHWQkc71oPzG+Mr4E5zD1/xzItWGTRmp1+9AdtX8MhLiz Z2UENinliEK+kcESDyo8/HcSapUMvq9c9B6zoRy6TlYsJgvC4O8ivo9jD+XVBuBqtBMV dUgjasJVV+V56+w4uXz5A86H2a35+JURgd8BYJ9ck0lXDxD8CMvhwHAKPKEAj+MOTlOn /QtA== X-Gm-Message-State: AJIora8NyFMycbHkQX1ZSLGM5A+FBpvcGoU8rSqjbI2YsbB41xzj3gBt 6PrbQFPbPnHjvYBbPsYhmQ/rqMXkGb0KnKV5DyxgmA== X-Google-Smtp-Source: AGRyM1t9WQkJ2cZEurotfSjnoPbzZYp5/Nfaaj9MoN5XpJDdUXwQY52sQWwNS/4aCPvBX7itJzpfPuNQwLRYKYUjF9M= X-Received: by 2002:a5d:664d:0:b0:21a:3b82:6bb2 with SMTP id f13-20020a5d664d000000b0021a3b826bb2mr15177634wrw.534.1656379910608; Mon, 27 Jun 2022 18:31:50 -0700 (PDT) MIME-Version: 1.0 References: <20220621125658.64935-1-songmuchun@bytedance.com> In-Reply-To: From: Yosry Ahmed Date: Mon, 27 Jun 2022 18:31:14 -0700 Message-ID: Subject: Re: [PATCH v6 00/11] Use obj_cgroup APIs to charge the LRU pages To: Roman Gushchin Cc: Muchun Song , Andrew Morton , Johannes Weiner , longman@redhat.com, Michal Hocko , Shakeel Butt , Cgroups , duanxiongchun@bytedance.com, Linux Kernel Mailing List , Linux-MM Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656379912; a=rsa-sha256; cv=none; b=y7Gs0btsjYfTlyMcSKv/iUxWQ3bkYrjO2iI0LliOeyHNBndFf7YB6DSxEeWqt/N9PGfT5n voh41IQLTzLvX+a1k3atQOAqsIL3VfebopH7SdOLxjc/nBKSBiQ+IrE66+4Ik+55ctEg3T RzUVxsx3hVMptPzx7yJSigY/G6Jx+Xk= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cKi7cvaj; spf=pass (imf02.hostedemail.com: domain of yosryahmed@google.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656379912; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C1kij541w9Z+pa/MZJLadURpUoXi65YxPa2CRYu92jA=; b=AZbZfwbMWmBhXxZcMmdsmEwv6Xb1D14ldT4QjSIMUVH9vdJMrh30b2/agmF9/gDEULBTW4 0yZwUck59ggvdxVK05J9JnykabyJemQe16fYSqSnNirbzLJ98a7jLTgvSCK0vqx29xpXx2 hZPGacALD5dGe89AQjBjw7G3BrNWxe4= X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 17A8780032 X-Rspam-User: X-Stat-Signature: 9b3hkipw3d3s6yom6d3hfi616qqeoesw Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cKi7cvaj; spf=pass (imf02.hostedemail.com: domain of yosryahmed@google.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1656379911-120327 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 27, 2022 at 6:24 PM Roman Gushchin wrote: > > On Mon, Jun 27, 2022 at 06:13:48PM +0800, Muchun Song wrote: > > On Mon, Jun 27, 2022 at 01:05:06AM -0700, Yosry Ahmed wrote: > > > On Mon, Jun 27, 2022 at 12:11 AM Muchun Song wrote: > > > > > > > > On Sun, Jun 26, 2022 at 03:32:02AM -0700, Yosry Ahmed wrote: > > > > > On Tue, Jun 21, 2022 at 5:57 AM Muchun Song wrote: > > > > > > > > > > > > This version is rebased on mm-unstable. Hopefully, Andrew can get this series > > > > > > into mm-unstable which will help to determine whether there is a problem or > > > > > > degradation. I am also doing some benchmark tests in parallel. > > > > > > > > > > > > Since the following patchsets applied. All the kernel memory are charged > > > > > > with the new APIs of obj_cgroup. > > > > > > > > > > > > commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects instead of pages") > > > > > > commit b4e0b68fbd9d ("mm: memcontrol: use obj_cgroup APIs to charge kmem pages") > > > > > > > > > > > > But user memory allocations (LRU pages) pinning memcgs for a long time - > > > > > > it exists at a larger scale and is causing recurring problems in the real > > > > > > world: page cache doesn't get reclaimed for a long time, or is used by the > > > > > > second, third, fourth, ... instance of the same job that was restarted into > > > > > > a new cgroup every time. Unreclaimable dying cgroups pile up, waste memory, > > > > > > and make page reclaim very inefficient. > > > > > > > > > > > > We can convert LRU pages and most other raw memcg pins to the objcg direction > > > > > > to fix this problem, and then the LRU pages will not pin the memcgs. > > > > > > > > > > > > This patchset aims to make the LRU pages to drop the reference to memory > > > > > > cgroup by using the APIs of obj_cgroup. Finally, we can see that the number > > > > > > of the dying cgroups will not increase if we run the following test script. > > > > > > > > > > This is amazing work! > > > > > > > > > > Sorry if I came late, I didn't follow the threads of previous versions > > > > > so this might be redundant, I just have a couple of questions. > > > > > > > > > > a) If LRU pages keep getting parented until they reach root_mem_cgroup > > > > > (assuming they can), aren't these pages effectively unaccounted at > > > > > this point or leaked? Is there protection against this? > > > > > > > > > > > > > In this case, those pages are accounted in root memcg level. Unfortunately, > > > > there is no mechanism now to transfer a page's memcg from one to another. > > > > > > > > > b) Since moving charged pages between memcgs is now becoming easier by > > > > > using the APIs of obj_cgroup, I wonder if this opens the door for > > > > > future work to transfer charges to memcgs that are actually using > > > > > reparented resources. For example, let's say cgroup A reads a few > > > > > pages into page cache, and then they are no longer used by cgroup A. > > > > > cgroup B, however, is using the same pages that are currently charged > > > > > to cgroup A, so it keeps taxing cgroup A for its use. When cgroup A > > > > > dies, and these pages are reparented to A's parent, can we possibly > > > > > mark these reparented pages (maybe in the page tables somewhere) so > > > > > that next time they get accessed we recharge them to B instead > > > > > (possibly asynchronously)? > > > > > I don't have much experience about page tables but I am pretty sure > > > > > they are loaded so maybe there is no room in PTEs for something like > > > > > this, but I have always wondered about what we can do for this case > > > > > where a cgroup is consistently using memory charged to another cgroup. > > > > > Maybe when this memory is reparented is a good point in time to decide > > > > > to recharge appropriately. It would also fix the reparenty leak to > > > > > root problem (if it even exists). > > > > > > > > > > > > > From my point of view, this is going to be an improvement to the memcg > > > > subsystem in the future. IIUC, most reparented pages are page cache > > > > pages without be mapped to users. So page tables are not a suitable > > > > place to record this information. However, we already have this information > > > > in struct obj_cgroup and struct mem_cgroup. If a page's obj_cgroup is not > > > > equal to the page's obj_cgroup->memcg->objcg, it means this page have > > > > been reparented. I am thinking if a place where a page is mapped (probably > > > > page fault patch) or page (cache) is written (usually vfs write path) > > > > is suitable to transfer page's memcg from one to another. But need more > > > > > > Very good point about unmapped pages, I missed this. Page tables will > > > do us no good here. Such a change would indeed require careful thought > > > because (like you mentioned) there are multiple points in time where > > > it might be suitable to consider recharging the page (e.g. when the > > > page is mapped). This could be an incremental change though. Right now > > > we have no recharging at all, so maybe we can gradually add recharging > > > to suitable paths. > > > > > > > Agree. > > > > > > thinking, e.g. How to decide if a reparented page needs to be transferred? > > > > > > Maybe if (page's obj_cgroup->memcg == root_mem_cgroup) OR (memcg of > > > > This is a good start. > > > > > current is not a descendant of page's obj_cgroup->memcg) is a good > > > > I am not sure this one since a page could be shared between different > > memcg. > > No way :) No way in terms of charging or usage? AFAIU a page is only charged to one memcg, but can be used by multiple memcgs if it exists in the page cache for example. Am I missing something here? > > > > > root > > / \ > > A B > > / \ \ > > C E D > > > > e.g. a page (originally, it belongs to memcg E and E is dying) is reparented > > to memcg A, and it is shared between C and D now. Then we need to consider > > whether it should be recharged. Yep, we need more thinging about recharging. > > This is why I wasn't sure that objcg-based reparenting is the best approach. > Instead (or maybe even _with_ the reparenting) we can recharge pages on, say, > page activation and/or rotation (inactive->inactive). Pagefaults/reads are > probably to hot to do it there. But the reclaim path should be more accessible > in terms of the performance overhead. Just some ideas. Thanks for chipping in, Roman! I am honestly not sure on what paths the recharge should occur, but I know that we will probably need a recharge mechanism at some point. We can start adding recharging gradually to paths that don't affect performance, reclaim is a very good place. Maybe we sort LRUs such that reparented pages are scanned first, and possibly recharged under memcg pressure. > > Thanks!