From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.3 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6854AC433E0 for ; Tue, 7 Jul 2020 15:51:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0A5FC2082F for ; Tue, 7 Jul 2020 15:51:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="L3SIjbQ2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0A5FC2082F Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 499976B0073; Tue, 7 Jul 2020 11:51:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 449006B0074; Tue, 7 Jul 2020 11:51:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 337AA6B0075; Tue, 7 Jul 2020 11:51:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0153.hostedemail.com [216.40.44.153]) by kanga.kvack.org (Postfix) with ESMTP id 16E816B0073 for ; Tue, 7 Jul 2020 11:51:44 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 823151EE6 for ; Tue, 7 Jul 2020 15:51:43 +0000 (UTC) X-FDA: 77011720086.20.snail71_2415ff426eb5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 1B5FA180C07A3 for ; Tue, 7 Jul 2020 15:51:43 +0000 (UTC) X-HE-Tag: snail71_2415ff426eb5 X-Filterd-Recvd-Size: 7848 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Tue, 7 Jul 2020 15:51:42 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id 9so50448349ljv.5 for ; Tue, 07 Jul 2020 08:51:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=y/kOBwO7OLhAG5/zoDioEbhDbIra0ifUwtyOcu+taVE=; b=L3SIjbQ2aGUMKNVOsgYJSH0RyGv1yqE3EspINaDxV4Qp8l0EecCKFOqwnxga/y+h8Y 2X0KH3FZOV8/IlyRauMigZhGEIdAaQWWZD+yMxaZDDJrf0UKEJCKQTztAwNoUxcOcoSr 8MLugsOwfz0qFWdozatId6q9p//k2M4rt6i+b82/4DZD+KliP8EFP5t2gEeqrOJX9hZL 5U+eSXspaw2k59P+sBud1CuBDW7zu5vUC6+vNCHxtZRxVveGPJaXdmkBhF1NZ9SqydwI elgQJT7e7qGxHtzdpoLU8GvAZlD/n7SQevJljznZY23clvyeaem0Qz6xKoorVTHKsPlf +xuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=y/kOBwO7OLhAG5/zoDioEbhDbIra0ifUwtyOcu+taVE=; b=XnmeiaYbU6cdcqyKCTnRusWf9M869B+aqpQYdrzU1UOvY1lWAmqSuK37smyxTWf86B 5noEAyfDw9T4KNs3rrBGLtHVtWYjH2ZyrQsYiLCDCE5Tt3k7lV43M5jSdJh+FQ6HFInb mT3equDyXcPkds0vl56NY/psT+OlIAQfJ1jXE2kdneTo8+TNMgPx5+KZAAHTFMDDqPw7 gmP4Am/5d6T9hZYaOIL9qScsZ0arx5s26tJBo92Xgcs2oAe/fD0VyhoF/2VExj3ClNci fSA3HKjmIncoZvo17V5A1Y8zTmrskfvsdDrqyM+YuG+Qv15EupQFfqAO6k2hkIrTdEM4 XvYg== X-Gm-Message-State: AOAM532vQ/kbCqCgt1tbeAKLo4n4y/hAjbghhHpJOVL4EptjDehBfL1q YwO8raKV4yTloGXNJY/vArf8UsAAB/oGO689/bQYhw== X-Google-Smtp-Source: ABdhPJyWaoNm3VNK+Rrjg/sAMM/2RSzMf89FEFqIRJwKBh4jHqzLzK0U/GjrgZx+sjtuR+WLH8QN9UOhGSB/bj4TUFY= X-Received: by 2002:a2e:8e36:: with SMTP id r22mr31066177ljk.77.1594137100635; Tue, 07 Jul 2020 08:51:40 -0700 (PDT) MIME-Version: 1.0 References: <20200702152222.2630760-1-shakeelb@google.com> <20200703063548.GM18446@dhcp22.suse.cz> <20200703155021.GB114903@carbon.dhcp.thefacebook.com> <20200706213404.GA152560@carbon.lan> In-Reply-To: <20200706213404.GA152560@carbon.lan> From: Shakeel Butt Date: Tue, 7 Jul 2020 08:51:29 -0700 Message-ID: Subject: Re: [RFC PROPOSAL] memcg: per-memcg user space reclaim interface To: Roman Gushchin Cc: Michal Hocko , Johannes Weiner , Yang Shi , David Rientjes , Greg Thelen , Andrew Morton , Linux MM , LKML , Cgroups Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 1B5FA180C07A3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jul 6, 2020 at 2:38 PM Roman Gushchin wrote: > > On Fri, Jul 03, 2020 at 09:27:19AM -0700, Shakeel Butt wrote: > > On Fri, Jul 3, 2020 at 8:50 AM Roman Gushchin wrote: > > > > > > On Fri, Jul 03, 2020 at 07:23:14AM -0700, Shakeel Butt wrote: > > > > On Thu, Jul 2, 2020 at 11:35 PM Michal Hocko wrote: > > > > > > > > > > On Thu 02-07-20 08:22:22, Shakeel Butt wrote: > > > > > [...] > > > > > > Interface options: > > > > > > ------------------ > > > > > > > > > > > > 1) memcg interface e.g. 'echo 10M > memory.reclaim' > > > > > > > > > > > > + simple > > > > > > + can be extended to target specific type of memory (anon, file, kmem). > > > > > > - most probably restricted to cgroup v2. > > > > > > > > > > > > 2) fadvise(PAGEOUT) on cgroup_dir_fd > > > > > > > > > > > > + more general and applicable to other FSes (actually we are using > > > > > > something similar for tmpfs). > > > > > > + can be extended in future to just age the LRUs instead of reclaim or > > > > > > some new use cases. > > > > > > > > > > Could you explain why memory.high as an interface to trigger pro-active > > > > > memory reclaim is not sufficient. Also memory.low limit to protect > > > > > latency sensitve workloads? > > > > > > I initially liked the proposal, but after some thoughts I've realized > > > that I don't know a good use case where memory.high is less useful. > > > Shakeel, what's the typical use case you thinking of? > > > Who and how will use the new interface? > > > > > > > > > > > Yes, we can use memory.high to trigger [proactive] reclaim in a memcg > > > > but note that it can also introduce stalls in the application running > > > > in that memcg. Let's suppose the memory.current of a memcg is 100MiB > > > > and we want to reclaim 20MiB from it, we can set the memory.high to > > > > 80MiB but any allocation attempt from the application running in that > > > > memcg can get stalled/throttled. I want the functionality of the > > > > reclaim without potential stalls. > > > > > > But reclaiming some pagecache/swapping out anon pages can always > > > generate some stalls caused by pagefaults, no? > > > > > > > Thanks for looking into the proposal. Let me answer both of your > > questions together. I have added the two use-cases but let me explain > > the proactive reclaim a bit more as we actually use that in our > > production. > > > > We have defined tolerable refault rates for the applications based on > > their type (latency sensitive or not). Proactive reclaim is triggered > > in the application based on their current refault rates and usage. If > > the current refault rate exceeds the tolerable refault rate then > > stop/slowdown the proactive reclaim. > > > > For the second question, yes, each individual refault can induce the > > stall as well but we have more control on that stall as compared to > > stalls due to reclaim. For us almost all the reclaimable memory is > > anon and we use compression based swap, so, the cost of each refault > > is fixed and a couple of microseconds. > > > > I think the next question is what about the refaults from disk or > > source with highly variable cost. Usually the latency sensitive > > applications remove such uncertainty by mlocking the pages backed by > > such backends (e.g. mlocking the executable) or at least that is the > > case for us. > > Got it. > > It feels like you're suggesting something similar to memory.high with > something similar to a different gfp flags. In other words, the > difference is only which pages can be reclaimed and which not. I don't > have a definitive answer here, but I wonder if we can somehow > generalize the existing interface? E.g. if the problem is with artificially > induced delays, we can have a config option/sysctl/sysfs knob/something else > which would disable it. Otherwise we risk ending up with many different kinds > of soft memory limits. > It is possible to achieve this functionality with memory.high with some config/sysctls e.t.c as you suggested but it can bloat and complicate the memory.high interface. I understand your concern with different kinds of soft memory limits but I see this as a simple interface (i.e. just trigger reclaim) that gives users the flexibility to define and (soft) enforce their own virtual limits on their jobs. More specifically this interface allows reclaiming from a job to keep the usage below some virtual limit which can correspond to some user defined metric. In my proactive reclaim example, the user defined metric is refault rates. Keep the usage of the job at a level where the refault rates are tolerable.