From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EEF7C433F5 for ; Fri, 1 Apr 2022 10:03:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3C408D0002; Fri, 1 Apr 2022 06:02:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEDA08D0001; Fri, 1 Apr 2022 06:02:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B3AA8D0002; Fri, 1 Apr 2022 06:02:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 8CF158D0001 for ; Fri, 1 Apr 2022 06:02:57 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 2052C81F5D for ; Fri, 1 Apr 2022 10:02:47 +0000 (UTC) X-FDA: 79307871174.15.45ADF7A Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf25.hostedemail.com (Postfix) with ESMTP id 9E012A0005 for ; Fri, 1 Apr 2022 10:02:46 +0000 (UTC) Received: by mail-pj1-f50.google.com with SMTP id h23-20020a17090a051700b001c9c1dd3acbso2052325pjh.3 for ; Fri, 01 Apr 2022 03:02:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=qxRwtOAESxeyz5WVUXRD0tS/GqKWWNmEkjZmJLerbDY=; b=jXM6XCXbGTW+340l5PT/qmsNmQAzjIlDKLRYi3fjrwdgvAE4JxyY9bPj4Je60Xc0DZ Ki/AuOfjLSzh4n4a1EXqNXEXRlk3uuWRMZVZTu/7vqM7kCedYF/9fcDz29UHnkoLMcsK rbz5HQ7JZLNAh03zo1jNtFV43U6VclXtsqCEoIoN+a21+LKtk+gr187R5ll0nWaMTB7Z JraAJt/VAFjl/WMST1MN8xIRObcxpA6PPQ0CIOwdZZxH8BQUCy7AVE4vidwka30BBurx S0q3YvWBKlQQMCmRtOCM1Nx1CgPwlvIVQbnbxiEetRcYiENhbdWKYeMOT0H6KHgfYiV7 tHuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=qxRwtOAESxeyz5WVUXRD0tS/GqKWWNmEkjZmJLerbDY=; b=Y3ifn36/oHYB8Fne8tC7REzCfQxEx/RXiBxiCBVj3DJRCrSskcin25bmXRg1CEckhL WW3DSPH3Y9iQZNX4eO1dD7u8Xjj5kBmTqWKbBeJac3Yg9Bb9YjYh0d+C/hA0XXQTviwV vaA1XmWi7XaLK1fGnfGZ1GYV823NSdP4+I0WdxDflh48/eSXaKM8pINxVbRQlesZLAbr 2x5Ni1umUeywlBYRjAg968gvrH9pcHG1pBs4an24L79S1P2X7m+/j8UkVE5ycXhVzJf5 qhae7dOw4Qk63xc/T6marb8MxnSWa1Gg9T1mQc0P4GxX++4WuIfsJkclZGsWPm3pmqj6 /Htg== X-Gm-Message-State: AOAM530z3VRJ8T3AdwowMnHE66t1vAh/uSRjXCmnIITonGD2EXTHmtPO f4gTLlxlYdy8Ah+NaqOqUovopO066pkMJ7UCFQNTMSEL/01Y0Q== X-Google-Smtp-Source: ABdhPJwtL7Qh3sI9nN/AEwuEQz3FhFuzBIwd156mZBBK3y+g1FSSIQ06sOcoy9fvFn0Wc2klt1gykhnkvPfMI3olp5c= X-Received: by 2002:a17:902:f24b:b0:154:de5:a24c with SMTP id j11-20020a170902f24b00b001540de5a24cmr9687863plc.162.1648807365285; Fri, 01 Apr 2022 03:02:45 -0700 (PDT) MIME-Version: 1.0 References: <20220331084151.2600229-1-yosryahmed@google.com> <2025405d-c32b-338a-b668-48b07a34e4ef@huawei.com> In-Reply-To: From: Yosry Ahmed Date: Fri, 1 Apr 2022 03:02:09 -0700 Message-ID: Subject: Re: [PATCH resend] memcg: introduce per-memcg reclaim interface To: Chen Wandun Cc: Johannes Weiner , Michal Hocko , Shakeel Butt , Andrew Morton , David Rientjes , Tejun Heo , Zefan Li , Roman Gushchin , cgroups@vger.kernel.org, linux-doc@vger.kernel.org, Linux Kernel Mailing List , Linux-MM , Jonathan Corbet , Yu Zhao , Dave Hansen , Wei Xu , Greg Thelen Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: kb4fkf3i4cd6ne3nyzc66udesfzs5abd Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jXM6XCXb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of yosryahmed@google.com designates 209.85.216.50 as permitted sender) smtp.mailfrom=yosryahmed@google.com X-Rspamd-Queue-Id: 9E012A0005 X-HE-Tag: 1648807366-568826 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Apr 1, 2022 at 2:49 AM Chen Wandun wrote: > > > > =E5=9C=A8 2022/4/1 17:20, Yosry Ahmed =E5=86=99=E9=81=93: > > On Thu, Mar 31, 2022 at 8:05 PM Chen Wandun wro= te: > >> > >> > >> =E5=9C=A8 2022/3/31 16:41, Yosry Ahmed =E5=86=99=E9=81=93: > >>> From: Shakeel Butt > >>> > >>> Introduce an memcg interface to trigger memory reclaim on a memory cg= roup. > >>> > >>> Use case: Proactive Reclaim > >>> --------------------------- > >>> > >>> A userspace proactive reclaimer can continuously probe the memcg to > >>> reclaim a small amount of memory. This gives more accurate and > >>> up-to-date workingset estimation as the LRUs are continuously > >>> sorted and can potentially provide more deterministic memory > >>> overcommit behavior. The memory overcommit controller can provide > >>> more proactive response to the changing behavior of the running > >>> applications instead of being reactive. > >>> > >>> A userspace reclaimer's purpose in this case is not a complete replac= ement > >>> for kswapd or direct reclaim, it is to proactively identify memory sa= vings > >>> opportunities and reclaim some amount of cold pages set by the policy > >>> to free up the memory for more demanding jobs or scheduling new jobs. > >>> > >>> A user space proactive reclaimer is used in Google data centers. > >>> Additionally, Meta's TMO paper recently referenced a very similar > >>> interface used for user space proactive reclaim: > >>> https://dl.acm.org/doi/pdf/10.1145/3503222.3507731 > >>> > >>> Benefits of a user space reclaimer: > >>> ----------------------------------- > >>> > >>> 1) More flexible on who should be charged for the cpu of the memory > >>> reclaim. For proactive reclaim, it makes more sense to be centralized= . > >>> > >>> 2) More flexible on dedicating the resources (like cpu). The memory > >>> overcommit controller can balance the cost between the cpu usage and > >>> the memory reclaimed. > >>> > >>> 3) Provides a way to the applications to keep their LRUs sorted, so, > >>> under memory pressure better reclaim candidates are selected. This al= so > >>> gives more accurate and uptodate notion of working set for an > >>> application. > >>> > >>> Why memory.high is not enough? > >>> ------------------------------ > >>> > >>> - memory.high can be used to trigger reclaim in a memcg and can > >>> potentially be used for proactive reclaim. > >>> However there is a big downside in using memory.high. It can pote= ntially > >>> introduce high reclaim stalls in the target application as the > >>> allocations from the processes or the threads of the application = can hit > >>> the temporary memory.high limit. > >>> > >>> - Userspace proactive reclaimers usually use feedback loops to decide > >>> how much memory to proactively reclaim from a workload. The metri= cs > >>> used for this are usually either refaults or PSI, and these metri= cs > >>> will become messy if the application gets throttled by hitting th= e > >>> high limit. > >>> > >>> - memory.high is a stateful interface, if the userspace proactive > >>> reclaimer crashes for any reason while triggering reclaim it can = leave > >>> the application in a bad state. > >>> > >>> - If a workload is rapidly expanding, setting memory.high to proactiv= ely > >>> reclaim memory can result in actually reclaiming more memory than > >>> intended. > >>> > >>> The benefits of such interface and shortcomings of existing interface > >>> were further discussed in this RFC thread: > >>> https://lore.kernel.org/linux-mm/5df21376-7dd1-bf81-8414-32a73cea45dd= @google.com/ > >>> > >>> Interface: > >>> ---------- > >>> > >>> Introducing a very simple memcg interface 'echo 10M > memory.reclaim'= to > >>> trigger reclaim in the target memory cgroup. > >>> > >>> > >>> Possible Extensions: > >>> -------------------- > >>> > >>> - This interface can be extended with an additional parameter or flag= s > >>> to allow specifying one or more types of memory to reclaim from (= e.g. > >>> file, anon, ..). > >>> > >>> - The interface can also be extended with a node mask to reclaim from > >>> specific nodes. This has use cases for reclaim-based demotion in = memory > >>> tiering systens. > >>> > >>> - A similar per-node interface can also be added to support proactive > >>> reclaim and reclaim-based demotion in systems without memcg. > >>> > >>> For now, let's keep things simple by adding the basic functionality. > >>> > >>> [yosryahmed@google.com: refreshed to current master, updated commit > >>> message based on recent discussions and use cases] > >>> Signed-off-by: Shakeel Butt > >>> Signed-off-by: Yosry Ahmed > >>> --- > >>> Documentation/admin-guide/cgroup-v2.rst | 9 ++++++ > >>> mm/memcontrol.c | 37 +++++++++++++++++++++= ++++ > >>> 2 files changed, 46 insertions(+) > >>> > >>> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/= admin-guide/cgroup-v2.rst > >>> index 69d7a6983f78..925aaabb2247 100644 > >>> --- a/Documentation/admin-guide/cgroup-v2.rst > >>> +++ b/Documentation/admin-guide/cgroup-v2.rst > >>> @@ -1208,6 +1208,15 @@ PAGE_SIZE multiple when read back. > >>> high limit is used and monitored properly, this limit's > >>> utility is limited to providing the final safety net. > >>> > >>> + memory.reclaim > >>> + A write-only file which exists on non-root cgroups. > >>> + > >>> + This is a simple interface to trigger memory reclaim in the > >>> + target cgroup. Write the number of bytes to reclaim to this > >>> + file and the kernel will try to reclaim that much memory. > >>> + Please note that the kernel can over or under reclaim from > >>> + the target cgroup. > >>> + > >>> memory.oom.group > >>> A read-write single value file which exists on non-root > >>> cgroups. The default value is "0". > >>> diff --git a/mm/memcontrol.c b/mm/memcontrol.c > >>> index 725f76723220..994849fab7df 100644 > >>> --- a/mm/memcontrol.c > >>> +++ b/mm/memcontrol.c > >>> @@ -6355,6 +6355,38 @@ static ssize_t memory_oom_group_write(struct k= ernfs_open_file *of, > >>> return nbytes; > >>> } > >>> > >>> +static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf= , > >>> + size_t nbytes, loff_t off) > >>> +{ > >>> + struct mem_cgroup *memcg =3D mem_cgroup_from_css(of_css(of)); > >>> + unsigned int nr_retries =3D MAX_RECLAIM_RETRIES; > >>> + unsigned long nr_to_reclaim, nr_reclaimed =3D 0; > >>> + int err; > >>> + > >>> + buf =3D strstrip(buf); > >>> + err =3D page_counter_memparse(buf, "", &nr_to_reclaim); > >>> + if (err) > >>> + return err; > >>> + > >>> + while (nr_reclaimed < nr_to_reclaim) { > >>> + unsigned long reclaimed; > >>> + > >>> + if (signal_pending(current)) > >>> + break; > >>> + > >>> + reclaimed =3D try_to_free_mem_cgroup_pages(memcg, > >>> + nr_to_reclaim - nr_recl= aimed, > >>> + GFP_KERNEL, true); > >> In some scenario there are lots of page cache, and we only want to > >> reclaim page cache, > >> how about add may_swap option? > > Thanks for taking a look at this! > > > > The first listed extension is an argument/flags to specify the type of > do you mean nbytes in memory_reclaim? it decide the amount of memory > to reclaim. > > one more argument such as may_swap can be add into memory_reclaim, and > pass this argument to try_to_free_mem_cgroup_pages in order to replace th= e > default "true" > > Thanks. I agree about the need for a may_swap or similar argument. In the commit message I list some possible extensions to this interface, and the first one is to add an argument to specify the type of memory we want to reclaim using the interface (anon, file, ..), which I think covers this use case. I just think we should add this in a separate patch as an extension. > > > memory that we want to reclaim, I think this covers this use case, or > > am I missing something? > > > >>> + > >>> + if (!reclaimed && !nr_retries--) > >>> + break; > >>> + > >>> + nr_reclaimed +=3D reclaimed; > >>> + } > >>> + > >>> + return nbytes; > >>> +} > >>> + > >>> static struct cftype memory_files[] =3D { > >>> { > >>> .name =3D "current", > >>> @@ -6413,6 +6445,11 @@ static struct cftype memory_files[] =3D { > >>> .seq_show =3D memory_oom_group_show, > >>> .write =3D memory_oom_group_write, > >>> }, > >>> + { > >>> + .name =3D "reclaim", > >>> + .flags =3D CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE, > >>> + .write =3D memory_reclaim, > >>> + }, > >>> { } /* terminate */ > >>> }; > >>> > > . >