From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDE4FC433E2 for ; Thu, 10 Sep 2020 16:10:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5B49E20C09 for ; Thu, 10 Sep 2020 16:10:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="v4rilgN8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5B49E20C09 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 79C7E6B0092; Thu, 10 Sep 2020 12:10:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74F0F6B0093; Thu, 10 Sep 2020 12:10:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63DA98E0005; Thu, 10 Sep 2020 12:10:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0244.hostedemail.com [216.40.44.244]) by kanga.kvack.org (Postfix) with ESMTP id 49C666B0092 for ; Thu, 10 Sep 2020 12:10:33 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id E62F7180AD80F for ; Thu, 10 Sep 2020 16:10:32 +0000 (UTC) X-FDA: 77247639504.29.hook57_2f10f60270e7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 8174318086CAC for ; Thu, 10 Sep 2020 16:10:32 +0000 (UTC) X-HE-Tag: hook57_2f10f60270e7 X-Filterd-Recvd-Size: 10744 Received: from mail-lf1-f67.google.com (mail-lf1-f67.google.com [209.85.167.67]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Sep 2020 16:10:31 +0000 (UTC) Received: by mail-lf1-f67.google.com with SMTP id w11so3904546lfn.2 for ; Thu, 10 Sep 2020 09:10:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=kQgWv8tP/mIsTIaHVB2pZNWYND0t5P9IhVZh51psRFQ=; b=v4rilgN8QHJIugdS3zS0eNDdG0npIlrnqrIbjqTj/ttxLW42RtR+GzpuyFpCHQWt5D uyEUfNGcSMoqzxO1I+k+YF8A8HDx/L4R+QXilCE4oLNqZHx7Vd9y3zDfmRsTRPc2fFqg YEGBEfpU03XmHqkWmMvgsVatFOtQaatdc9jKQS9BbxxenINMx2NLxDvculChxaEUeNXg 3I1Mur4xaujZQzLrOsFwxMA3B0p+UgmXqs1coDdaO7ZAEL4xJ/aSVmKOQbHsZpkEReeI mx3Or9pu5gtiTmey8HgXRd7MNtBiOIkZ8N+kQB6UnYoy5vTvI5PdOvHDKPCFPal0lbUP aNdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=kQgWv8tP/mIsTIaHVB2pZNWYND0t5P9IhVZh51psRFQ=; b=C0qHxKIon22pwzW9040M6NXC9EXflBixZVlDmPWfETY7NMUa6hX+f5WSpcbuweYVdi v6KwpKyWaD//B6lqfDSHq5dZcw6ABLhjBRV6RRqmt9PP3YciXdwQRzLW3y1L/WnxwNlJ 4gl1BHGLWhOT9Tsrj9b+iEax3TAv6PM3k0/9o6XdKDfeBfXuuZiqmTzc5XdN7Pj/aT5k 3xRz576zegke1SXyGMAJ4cFhe57Jav17Rco/v/SSAeaLhktM6XOIu/n9VQzSI7ip/luh 0b3A3rJXjVcn10uu97pUuFW5rxxTBAENOceccQtD/lc6H6BiqQXosKogGgDd/qT72qXU ClPA== X-Gm-Message-State: AOAM530XJIoQBoQzVuMyW9+P42QcxCDv84J3BSmGnb33x6zUHgjKgL4j Ne81u6p4TQ/c+T6nm7bFp31NjbrNDXgSwTptErVUzw== X-Google-Smtp-Source: ABdhPJzijWg3vU9ZGbZG+XZJK8d4PmQ5H9p4dDg/+jvyg8Egp0XxgKSmR90qRecpk+XJ1GkepMOB+U2QDWHZbX3NBAA= X-Received: by 2002:a19:c8c6:: with SMTP id y189mr4615627lff.125.1599754229862; Thu, 10 Sep 2020 09:10:29 -0700 (PDT) MIME-Version: 1.0 References: <20200909215752.1725525-1-shakeelb@google.com> <20200910063656.25038-1-sjpark@amazon.com> In-Reply-To: <20200910063656.25038-1-sjpark@amazon.com> From: Shakeel Butt Date: Thu, 10 Sep 2020 09:10:18 -0700 Message-ID: Subject: Re: [PATCH] memcg: introduce per-memcg reclaim interface To: SeongJae Park Cc: Johannes Weiner , Roman Gushchin , Michal Hocko , Yang Shi , Greg Thelen , David Rientjes , =?UTF-8?Q?Michal_Koutn=C3=BD?= , Andrew Morton , Linux MM , Cgroups , LKML Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 8174318086CAC X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Sep 9, 2020 at 11:37 PM SeongJae Park wrote: > > On 2020-09-09T14:57:52-07:00 Shakeel Butt wrote: > > > Introduce an memcg interface to trigger memory reclaim on a memory cgroup. > > > > Use cases: > > ---------- > > > > 1) Per-memcg uswapd: > > > > Usually applications consists of combination of latency sensitive and > > latency tolerant tasks. For example, tasks serving user requests vs > > tasks doing data backup for a database application. At the moment the > > kernel does not differentiate between such tasks when the application > > hits the memcg limits. So, potentially a latency sensitive user facing > > task can get stuck in high reclaim and be throttled by the kernel. > > > > Similarly there are cases of single process applications having two set > > of thread pools where threads from one pool have high scheduling > > priority and low latency requirement. One concrete example from our > > production is the VMM which have high priority low latency thread pool > > for the VCPUs while separate thread pool for stats reporting, I/O > > emulation, health checks and other managerial operations. The kernel > > memory reclaim does not differentiate between VCPU thread or a > > non-latency sensitive thread and a VCPU thread can get stuck in high > > reclaim. > > > > One way to resolve this issue is to preemptively trigger the memory > > reclaim from a latency tolerant task (uswapd) when the application is > > near the limits. Finding 'near the limits' situation is an orthogonal > > problem. > > > > 2) Proactive reclaim: > > > > This is a similar to the previous use-case, the difference is instead of > > waiting for the application to be near its limit to trigger memory > > reclaim, continuously pressuring the memcg to reclaim a small amount of > > memory. This gives more accurate and uptodate workingset estimation as > > the LRUs are continuously sorted and can potentially provide more > > deterministic memory overcommit behavior. The memory overcommit > > controller can provide more proactive response to the changing behavior > > of the running applications instead of being reactive. > > > > Benefit of user space solution: > > ------------------------------- > > > > 1) More flexible on who should be charged for the cpu of the memory > > reclaim. For proactive reclaim, it makes more sense to centralized the > > overhead while for uswapd, it makes more sense for the application to > > pay for the cpu of the memory reclaim. > > > > 2) More flexible on dedicating the resources (like cpu). The memory > > overcommit controller can balance the cost between the cpu usage and > > the memory reclaimed. > > > > 3) Provides a way to the applications to keep their LRUs sorted, so, > > under memory pressure better reclaim candidates are selected. This also > > gives more accurate and uptodate notion of working set for an > > application. > > > > Questions: > > ---------- > > > > 1) Why memory.high is not enough? > > > > memory.high can be used to trigger reclaim in a memcg and can > > potentially be used for proactive reclaim as well as uswapd use cases. > > However there is a big negative in using memory.high. It can potentially > > introduce high reclaim stalls in the target application as the > > allocations from the processes or the threads of the application can hit > > the temporary memory.high limit. > > > > Another issue with memory.high is that it is not delegatable. To > > actually use this interface for uswapd, the application has to introduce > > another layer of cgroup on whose memory.high it has write access. > > > > 2) Why uswapd safe from self induced reclaim? > > > > This is very similar to the scenario of oomd under global memory > > pressure. We can use the similar mechanisms to protect uswapd from self > > induced reclaim i.e. memory.min and mlock. > > > > Interface options: > > ------------------ > > > > Introducing a very simple memcg interface 'echo 10M > memory.reclaim' to > > trigger reclaim in the target memory cgroup. > > > > In future we might want to reclaim specific type of memory from a memcg, > > so, this interface can be extended to allow that. e.g. > > > > $ echo 10M [all|anon|file|kmem] > memory.reclaim > > > > However that should be when we have concrete use-cases for such > > functionality. Keep things simple for now. > > > > Signed-off-by: Shakeel Butt > > --- > > Documentation/admin-guide/cgroup-v2.rst | 9 ++++++ > > mm/memcontrol.c | 37 +++++++++++++++++++++++++ > > 2 files changed, 46 insertions(+) > > > > diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst > > index 6be43781ec7f..58d70b5989d7 100644 > > --- a/Documentation/admin-guide/cgroup-v2.rst > > +++ b/Documentation/admin-guide/cgroup-v2.rst > > @@ -1181,6 +1181,15 @@ PAGE_SIZE multiple when read back. > > high limit is used and monitored properly, this limit's > > utility is limited to providing the final safety net. > > > > + memory.reclaim > > + A write-only file which exists on non-root cgroups. > > + > > + This is a simple interface to trigger memory reclaim in the > > + target cgroup. Write the number of bytes to reclaim to this > > + file and the kernel will try to reclaim that much memory. > > + Please note that the kernel can over or under reclaim from > > + the target cgroup. > > + > > memory.oom.group > > A read-write single value file which exists on non-root > > cgroups. The default value is "0". > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > > index 75cd1a1e66c8..2d006c36d7f3 100644 > > --- a/mm/memcontrol.c > > +++ b/mm/memcontrol.c > > @@ -6456,6 +6456,38 @@ static ssize_t memory_oom_group_write(struct kernfs_open_file *of, > > return nbytes; > > } > > > > +static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, > > + size_t nbytes, loff_t off) > > +{ > > + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); > > + unsigned int nr_retries = MAX_RECLAIM_RETRIES; > > + unsigned long nr_to_reclaim, nr_reclaimed = 0; > > + int err; > > + > > + buf = strstrip(buf); > > + err = page_counter_memparse(buf, "", &nr_to_reclaim); > > + if (err) > > + return err; > > + > > + while (nr_reclaimed < nr_to_reclaim) { > > + unsigned long reclaimed; > > + > > + if (signal_pending(current)) > > + break; > > + > > + reclaimed = try_to_free_mem_cgroup_pages(memcg, > > + nr_to_reclaim - nr_reclaimed, > > + GFP_KERNEL, true); > > + > > + if (!reclaimed && !nr_retries--) > > + break; > > Shouldn't the if condition use '||' instead of '&&'? I copied the pattern from memory_high_write(). > I think it could be > easier to read if we put the 'nr_retires' condition in the while condition as > below (just my personal preference, though). > > while (nr_reclaimed < nr_to_reclaim && nr_retires--) > The semantics will be different. In my version, it means tolerate MAX_RECLAIM_RETRIES reclaim failures and your suggestion means total MAX_RECLAIM_RETRIES tries. Please note that try_to_free_mem_cgroup_pages() internally does 'nr_to_reclaim = max(nr_pages, SWAP_CLUSTER_MAX)', so, we might need more than MAX_RECLAIM_RETRIES successful tries to actually reclaim the amount of memory the user has requested. > > Thanks, > SeongJae Park